Interacting with a web page, searching, drop-downs, buttons

[beknoll]

Greetings all,

I'm new to Lisp, but reading through 'Land of Lisp', and 'Practical Common Lisp'.

I teach art at a high school and we take attendance everyday. It's a pain and all mouse-based navigation.

I'd like to write a function like this:
CL_USER>(absent '(AB_CD_EF...)) ;;;where the letters are the first and last initial of the student

The web page displays the student's name and then a drop-down box next to it with multiple options (e.g., absent, field trip, etc---absent is the 2nd(?) option. The first is a blank.).

I'd like to find the element/student's name, move to the box closest to it, hit either 'a' or change the drop-down to the second option, then send a 'tab' to be ready to loop to the next student.

I've checked out DRAKMA, Parenscript, Javascript, the DOM Object Model and generally tried to find POST/GET commands that I might use. But it's all still a bit Greek to me.

Any direction for further research would be great. Thanks.

----
Web page


HTML Dev Tool view (limited)

[pjstirling]

Hmm

My first reading of your post assumed you want to write a web-app, that runs on your own machine, that then connects to your school's server in response to your input, because you mentioned the DOM, and parenscript, and because the REPL won't give you things like fancy completion without quite a bit of effort, but perhaps you are hoping for some sort of ncurses style UI?

Proper answer to your question will depend a lot on your goals:

It doesn't sound like you need persistence, you just want a sort of proxy that provides a better UI, in which case my default approach to that would be an electron app written mostly in parenscript and react.

However!

If the goal is a first CL project to get a flavour of things, then that approach is probably not what you want, because that route is mostly spent dealing with an improved surface-syntax of javascript rather than 'real' CL.

Thoughts?

[beknoll]

pjstirling,

Thanks for the reply!

My mentioning of DOM and Parenscript I included to show that I was trying to find a solution first on my own and not asking the community to solve the problem for me.

Electron looks really cool (watched the vid) and I guess it could be a desktop app made like a web app.

I was really just interested in learning to write and writing a Lisp program that could interact with a webpage sort of like I would (click boxes and things). I want to translate what I do manually to a more command-line sort of interface. Rather than having to click on the box next to each student, select from a dropdown, hit tab, then do the same for every other student who's absent or tardy, I could just type in one command with vars that would handle all of it. (CL-USER>(absent-today '(AB CD EF)). Then this would mark 'Alfonso Bufo', 'Claire Danes, and 'Ephreet Fabu' absent by automating all the other stuff.

You mentioned below, though, that the REPL wouldn't be the best place for auto-completing students' names based on their initials. So the program definitely wouldn't have to run directly from the REPL. I could call up a new command-line sort of window.

In my research, it looked like I'd need Javascript and the DOM to scrape the student names from the page and Javascript to interact with the schools page through POST and GET. Parenscript I thought I'd use to stay in Lisp instead of using Javascript directly.

It would just be a program for me, so I don't want it to have any sort of clickable or even navigable interface except via keyboard commands. So more like a DOS or UNIX command line, yeah. Probably not so much ncurses (which I wasn't directly familiar with but looked up) as that looks like it's trying to be a proto-GUI that requires arrow commands and such.

And I don't have access to the school's server except as html serving up the pages. I mean, I log in to the site so might include in the program an ability to do this automatically, but I'm not a tech at the school and don't have admin rights or direct server access.

I don't know if this clarifies, but I hope so.

Should I invest some time into using electron and react? I'm a novice, as I mentioned, but know that for what I want to do personally I'll need to interact with webpages. My next project is to get two grading systems that I usually work with in the browser (ProgressBook and Canvas) to keep grades synced when I update one of them. It's such a pain to export and import to each of my 6 classes for every project my art kids do! I figure there's a programming solution. And that I can find it if I'm persistent enough.

Thanks again for taking the time to read/respond.

[pjstirling]

You can certainly do this purely in lisp, it's just that you will need
to do more work (or use more libraries)

drakma will let you fetch a web page, and submit forms, so you
will have a function that does the login, and returns the cookie jar
so that you stay logged in for the session.

You again use drakma passing the cookie jar to load the page of
attendance stuff for a particular class, this will be a binary blob

cl-html5-parser will let you parse the downloaded web page into
a kind of DOM that can be accessed from lisp. (I've got a small
library that transforms that DOM tree into something slightly more
lispy). By scraping this DOM tree you will be able to build the list
of initials for students into some data structure (however you will
have edge cases when a class has 2 people with the same initials,
birthday paradox!)

You'll need a function that takes the list of initials, and then loops
the text input thing asking for stuff with the CL function
READ-LINE

Finally you use drakma again to upload a form in the style that
the server expects

The text input function will be somewhat complicated, because you will
have to handle a lot of stuff manually, e.g. what about typos (no
student matched XZ!), keeping track of whether a student has been
entered more than once (maybe you made a mistake the first time),
keeping track of what students haven't been handled at all yet, was
your input a sentinel that indicates you want to stop, etc...

Some of these things can be skipped perhaps, but that turns an
annoying programming problem that is handled once, into an annoying
problem that you risk having every day

On the electron issue:

I've written READ-LINE style UI, and web-app style UI, and I generally
consider the improvement in usability vs cost of implementation a bit
better in web-app, you can mostly get things like skipping backward
and forward for free in web-apps, whereas it can be a pain in
READ-LINE, you get copy/paste, and free cross-platform

Normally if you want to use a web-app style UI you would need a server
to serve it from, but electron lets you do it as soemthing you can run
purely on your machine. The way it provides access to native platform
stuff that would fall foul of the security sandbox in browsers is also
not to be sniffed at.

On React:

React is nice because it makes efficient a style of building web-apps
where your UI modifies a model and then you build a brand new UI from
the new model state (rather than doing the DOM surgery to transform
the old one into the new one). For simpler apps it is maybe not so
worthwhile, but if you want to handle things like enabling and
disabling controls in response to model conditions it is quite good.