Tool of Thought

APL for the Practical Man

Threading the HTMLRenderer

September 22, 2022

The HTMLRenderer supports web sockets, which gives us two-way asynchronous communication between the browser and the APL session. The browser can send a message to APL, and APL can send a message to the browser. However, in neither case is the sender waiting for a reply from the recipient. Much functionality can be achieved this way. However, there are at least two cases when, from APL, we want to send a message to the HTMLRenderer and wait for a response. The first is simply to execute a little general JavaScript. We might want to execute '2+2' or get the innerHTML from an element, or get the HTML of the entire page. This should work anytime, anywhere, and specifically it must work under the event handler of the WebSocketReceive event, which is exactly where it gets tricky.

The second is when, from APL, we want to fire an event in the HTMLRenderer which has been wired with a event handler that calls back into APL. APL handles the message, presumably changing some state, and perhaps sending a message back to the HTMLRenderer, changing some state there. In this case we want to be able to wait in APL for all this to happen, and then verify the state changes. In other words, we want to test our HTMLRenderer GUI code in a easy, linear fashion, in one APL process. This post attempts to explain an attempt to implement this synchronous behavior for these two cases.

The ExecutJavaScriptSync function executes a snippet of JavasScript and waits for a response:

          ExecuteJavaScriptSync←{
     ⍝ ⍺ ←→ Document
     ⍝ ⍵ ←→ Javascript
     ⍝ ← ←→ Result                                                     
     c←'"',⍵,'"'
     j←'execCode(',(⍕⎕TID),',',c,')'
     _←⍺ ExecuteJavaScript j
     ⎕TGET ⎕TID
 }

        

The left argument is the APLDOM object, the right argument is a string of JavaScript. The thread ID is used as a message identifier.

There is also a cover function that executes a string of javascript inside a specfic element:

          ExecuteOnElementSync←{
     ⍝ ⍺ ←→ Element
     ⍝ ⍵ ←→ Javascript
     ⍝ ← ←→ Result
     i←TagIndex ⍺
     t←⍺.Tag
     c←'document.getElementsByTagName'
     c,←'(''',t,''').item(',(⍕i),').',⍵,';'
     ⍺.Document ExecuteJavaScriptSync c
 }

        

The ExecuteJavaScript function just sends the data over the socket:

          ExecuteJavaScript←{
     ⍝ ⍺ ←→ Document
     ⍝ ⍵ ←→ JavaScript
     ⍺.HTMLRenderer.WebSocketSend ⍺.HTMLRenderer.WebSocketID ⍵ 1 1
 }

        

Note that we do not make use of the HTMLRenderer.ExecuteJavaScript method.

Once the message is sent, we wait for a token ( ⎕TGET ⎕TID ). Meanwhile, in the HTMLRenderer, the following JavaScript is executed on receipt of the message from APL:

            function execCode(id,code) { 
         const b = {Event: "SJSR", id: id, result: eval(code) }; 
         const m = JSON.stringify(b); 
         ws.send (m)
}      

        

This simply evaluates the code and sends the result and the id back to APL. It also invents and returns an event named "SJSR" for synchronous javascript result .

Back in APL, we are waiting for and processing messages from the HTMLRenderer via the HTMLRenderer's WebSocketReceive event:

          OnWebSocketReceive←{
     h←⊃⍵
     c←⎕JSON 3⊃⍵
     c.Event≡'SJSR':c.result ⎕TPUT c.id
     d←h.Document
     q←d.StopPropagation
     d.StopPropagation←0
     q:0
     c.HTMLRenderer←h
     j←c.TargetIndex
     k←c.CurrentTargetIndex
     te ce←(Elements d)[j k]
     c.Target←te
     c.CurrentTarget←ce
     h.LastTID←h.LastTID HandleRequest&c
     0
 }

        

This function must handle messages that are instigated by events in the browser, like clicking a button, as well as messages responding to synchronous calls from APL (the "SJSR" event).

We must allow OnWebSocketReceive to fire continuously without waiting for the event handler in APL (for the click event on a button, say) to complete. Otherwise if the event handler itself sends a synchronous request back to the browser, the system will deadlock, waiting for a response that will never arrive, as OnWebSocketReceive can never fire as it is waiting for the event handler to complete, which is waiting for the response to the synchronous request. Thus we need to thread something, somewhere. But this introduces a new problem. The moment we thread, we allow all browser events to be processed concurrently, when we only want SJSR events processed concurrently with respect to all other events. All events initiated by the browser must be processed sequentially and exclusively, unless of course explicitly threaded somewhere in their own handlers.

It is possible to thread OnWebSocketReceive itself, but this only makes it significantly more difficult to queue the events. Instead we thread at the very end where we call HandleRequest which waits for the previous request to complete before executing:

          HandleRequest←{
     _←{6::0 ⋄ ⎕TSYNC ⍵}⍺
     _←(⍎⍵.CurrentTarget⍎'On',⍵.Event)⍵
     ~EventToken∊⎕TREQ ⎕TNUMS:0
     ⍵ ⎕TPUT EventToken
 }

        

Because OnWebSockReceive is single-threaded, we are guaranteed that LastTID is updated before the next message is received, and events processed in the proper order.

If the event is "SJSR", that is if the event is the result of a synchronous JavaScript request, we put a token whose value is the result into the pool, signaling to the waiting thread that it now has its requested result. The second case for synchronous behavior is testing. We want to be in APL and initiate an event in the HTMLRenderer that in turn sends a message to APL that does some processing. When this processing is complete, we want to verify some state change. Let's look at a sample test function:

          TestDecrement←{
     d←⍵.Document
     r←d A.ElementById'result'
     v←⍎⊃r.Content
     b←d A.ElementById'decrement'
     _←b A.FireEventAndWait'click'
     (v-1)≠⍎⊃r.Content
 }

        

This function first inspects and saves the content of an element, then fires a click event in the HTMLRenderer and waits for the event handler to complete, and finally compares the new value in the element to the expected value. The FireEventAndWait function is:

          FireEventAndWait←{
     ⍝ ⍺ ←→ Element
     ⍝ ⍵ ←→ Event
     ⍺=0:0
     _←⍺ ExecuteOnElement ⍵,'()'
     ⎕TGET EventToken
 }

        

The EventToken is just a constant. Test functions MUST run in their own thread, separate from the thread the HTMLRenderer is using to process WebSocketReceive events, otherwise no events are processed as we sit waiting for EventToken . What now happens as we are waiting to get EventToken ? If we review the last couple of lines of HandleRequest above we see that after an event is processed if there is any thread waiting on an EventToken, then it is supplied:

               ~EventToken∊⎕TREQ ⎕TNUMS:0
     ⍵ ⎕TPUT EventToken 

        

The test framework can thread itself, so all of the tests run in a new thread:

          RunTests←{
     ⍝ ⍺ ←→ [Namespace of tests]
     ⍝ ⍵ ←→ HTMLRenderer
     ⍺←⊃⎕RSI
     ⎕TID=0:⍺ ∇&⍵
     s h←⍺ ⍵
     c←'Passed' 'Failed' 'Broken' 'N/A' 'Disabled'
     r←{
         b←{0::2 ⋄ (s⍎⍵)h}⍵
         b⊣⎕←⍵,': ',⍕b⊃c
     }¨'T's.⎕NL ¯3
     n←¯1+{≢⍵}⌸r,⍨⍳≢c
     ⍉↑(c,⊂'Total')(n,+/n)
 }

        

The only reason to thread the event handler in OnWebSocketReceive is to allow for synchronous JavaScript from within the event handler. If synchronous JavaScript is not needed, the & and the ⎕TSYNC may just be removed, and all event handlers run in thread 0, for what that is worth. Tests need to be threaded regardless, and as a byproduct this allows synchronous JavaScript to run in tests even when not threading the event handler.

A Document Object Model in APL

September 21, 2022

With a web browser now effectively built into the interpreter, constructing and manipulating HTML and the DOM in a simple and consistent way has never been more important. Explicitly catenating strings is no way to go through life. Lots of (my) old code looks like:

                 '<h1>',t,'</h1>'

        

or barely better we might define tag as:

                  tag←{'<',⍺,'>',⍵,'</',⍺,'>'}

        

And then tag this, that and the other, which all has to be catenated up. And then we enhance tag to take an argument with attributes. It doesn't get better. And when we are done, we have a horrific string, probably nested in various places by mistake, of hopefully valid HTML that we cannot manipulate in any meaningful way.

Consider instead a single simple function New :

             New←{
     ⍝ ⍺ ←→ [Parent, 0 - no parent]
     ⍝ ⍵ ←→ Tag [Content [Attributes]]
     ⍝ ← ←→ Element
     ⍺←0
     s←IsString ⍵
     t c a←s⊃(3↑⍵,⊂⍬)(⍵''⍬)
     e←⎕NS''
     e.Tag←,t
     e.Parent←⍺
     e.Content←''
     _←a SetAttributes e
     e⊣Add/⍺ e c
 }

        

We can now create an object for an element, and produce HTML from it:

                h←New 'h1' 'My Title'
      DOM2HTML h
<h1>My Title</h1>

        

Attributes may be specified by assignment:

                h.class←'chapter'
      DOM2HTML h
<h1 class="chapter">My Title</h1>

        

Elements can be created as children:

                d←New 'html'
      b←d New 'body'
      m←b New 'main'
      DOM2HTML d
<!DOCTYPE html>
<html>
  <body>
    <main></main>
  </body>
</html>

        

Content may be directly assigned:

                m.Content←h
      DOM2HTML d
<!DOCTYPE html>
<html>
  <body>
    <main>
      <h1 class="chapter">My Title</h1>
    </main>
  </body>
</html>
~~~      

        

Producing HTML without explicit catenation is only a small part of the benefit. Consider the function Elements :

          Elements←{
     ⍝ ⍵ ←→ Element
     ⍝ ← ←→ Vector of ⍵ and all sub elements
     326≠⎕DR ⍵:⍬
     ⍵,{c←{0=≢⍵:⍬ ⋄ ⍵/⍨326=⎕DR¨⍵}⍵.Content
         0=≢c:⍬
         ⊃,/c,¨∇¨c}⍵
 } 

        

This traverses the DOM and yields a simple vector of all the elements:

                e←Elements d
      e.Tag
┌────┬────┬────┬──┐
│html│body│main│h1│
└────┴────┴────┴──┘

        

With this function in hand, various covers like ElementByID or ElementsByTag are trivial to write. Now its easy to find all the h1 headers and change, say, their class.

Tables are especially painful to deal with without the proper tooling. With a few helper functions to extract cells, we can manipulate HTML tables like an array programmer should:

               t←NewTable ⍕¨3 4⍴⍳12
     c←BodyCells t
     c[;2].class←⊂'char'
     DOM2HTML t
<table>                       
  <tbody>                     
    <tr>                      
      <td>0</td>              
      <td>1</td>              
      <td class="char">2</td> 
      <td>3</td>              
    </tr>                     
    <tr>                      
      <td>4</td>              
      <td>5</td>              
      <td class="char">6</td> 
      <td>7</td>              
    </tr>                     
    <tr>                      
      <td>8</td>              
      <td>9</td>              
      <td class="char">10</td>
      <td>11</td>             
    </tr>                     
  </tbody>                    
</table>                      
~~~ 

        

Finally we can take HTML and get back the DOM:

                 html←DOM2HTML t
       t2←HTML2DOM html
       html≡DOM2HTML t2
1

        

Where HTML2DOM is:

          HTML2DOM←{
     ⍝ ⍵ ←→ HTML
     ⍝ ← ←→ DOM
     0=≢⍵:⍬
     {⍵⌷⍨⍳1=⍴⍵}0{
         m←⍵
         0=≢m:⍺
         b←m[;0]=0
         p←⍺{⍺ New 3↑1↓⍵}¨↓b⌿m
         m[;0]-←1
         p⊣p ∇¨1↓¨b⊂[0]m
     }⎕XML ⍵
 }

        

Once you start creating HTML this way, patterns arise and utility functions fall out naturally. These utilities return content that can be manipulated and injected anywhere.

On Control Structures

August 1, 2022

The latest episode of Array Cast is on control structures. As usual, it is an excellent episode, covering history and all the options in the various array languages. However, the panelists raised a number of interesting questions which were not fully examined, let alone answered - understandable, given how much material was covered and the time constraints. One question was why use control structures at all?

One of the main arguments made in favor of using control structures was expediency . While this argument was meant to be proscriptive, it is definitely descriptive: it is obvious from inspecting large APL code bases that make heavy use of control structures. But expediency is just another word for a shortcut. We don't have time to do it right, so we take the fast and easy way out. As tempting as this always is, it is almost never the right approach.

Expediency piles on the technical debt. When hitting a bug, instead of refactoring the code, and perhaps extracting a new function or two, we just if-then-else or trap around the problem. Functions lose their focus, they get longer and longer, variables proliferate, control structures get nested, and nested, and nested again.

Note that the issue here is not searching for an array-based solution, which, as noted in the podcast, may be difficult (or even impossible) and may not be worth the time. The issue is general refactoring, important in any language or paradigm. When there is a function that appears to require lots of control structure statements, it is often possible to rearrange things and factor out the control structures. A new function or two may need to be extracted to cleanly eliminate some control structures. This effort, and make no mistake, it is indeed effort, is well worth it.

By eliminating control structures, one gains a deeper understanding of the problem, and the resulting code is shorter, more direct, and easier to debug, maintain and enhance. And by extracting more and more functions a domain-specfic English vocabulary arises and separates itself from the APL primitives. The code can become self-documenting. In the end we may still need some control structures, but they are separated and are above our APL primitive code. We don't have a :If keyword followed by a welter of primitives, a function train, and an embedded assignment. Functions are written at the same level of abstraction. Functions are mostly written in English, or mostly written in APL. As was noted in the podcast, it has been suggested that control structures are sort of outside the language of APL. If it is a good idea that we code at the same level of abstraction, it follows that control structures should not be mixed with the heavy use of primitives.

Most importantly, control structures provide an easy way to avoid naming things. Refactoring code to remove control structures requires naming new functions. Naming things is important. Naming is also hard under the best of circumstances. When looking at a large rambling function is can be impossible to find any section that is nameable - without rewriting everything. When a program is a single large function full of control structures, a fix or enhancement can be made with little thought to where it goes, precisely because there is no particular place for anything. There is only one name, the name of the big function. It's like a large messy room with no closet, chest of drawers, or desk. Anything can just get thrown in the room with no concern for where it is. This is chaos.

In a program with many small well-named functions, the place for a change must be carefully considered. There may be no good place for the change, and a new function may be required. Or the change may go in a particular function, expanding or changing its duties and making its name inappropriate. A change can easily require multiple function name changes and the introduction of new functions. This takes work. This program is like a large neat room with appropriate storage; everything has a place, and everything in its place. The dirty shirt has a place in the hamper, the book has a place on the desk. If the room acquires a lot of books, and the desk becomes unmanageable, a bookshelf is installed. This is order. Bringing order out of chaos requires naming. Keeping things in order requires renaming.

In addition to the question of why use control structures at all, the panelists pondered if it is possible to write large scale applications without them. These two questions are related, and if the answer to the first is primarily "expediency", the answer to the second is definitely "yes".

More posts...