Backend babbling about Boskone
A month after working Arisia 2021, I helped with the also-virtual Boskone 2021. It was an opportunity to directly compare several methodologies -- both for con-running, and API-level integration with Zoom via Grenadine. It's all in the long email I fired off to techno-fandom the week afterward. Date: Wed, 17 Feb 2021 14:28:43 Subject: a bolus about Boskone To: <techno-fandom list> Some of us did the virtualized Boskone last weekend, and I've got some bits of commentary from that. Longish, as usual. First, I wasn't actually doing any session hosting. I wound up being part of the back-end crew, helping to get the running structure configured up and even helping with some live training, and that turned out to be a fair amount of work. I also had my first go-round with Grenadine, which I have decidedly mixed feelings about in retrospect. One really nice thing we had, which we didn't think to set up for Arisia, was a tech backchannel Zoom that we could just leave running all day. Even if you don't look at it while concentrating on the stuff you're actually working on, it's good to just be able to *talk* back and forth, just like being on comm in a ballroom, bringing more of our senses into the process of running stuff. Being able to say "okay, *that* is now reconfigured" while hitting the <return> to make it happen is a lot faster than having to switch applications, go to some chat channel, and type that in, before rocketing on to the next task. So a working "tech audio" definitely has high value, and it doesn't have to be complicated as long as it can just stay running on its own. It can be a spare Zoom, or on Discord-based cons it could be any of a dozen audio channels we already know about. I had mine just running on the tablet next to me, so it didn't interfere at all with what I was doing on the main box I was running things with. It can be the live work channel when things are hopping, and the idle snark channel in between. It had a pleasantly "NOC-like" mission-control feel about it, almost like we were all in the same room. Having become reasonably fluent with both Slack and Discord at this point, I can say I *far* favor Discord for both staff comms and social spaces for event attendees. [Boskone's Slack wasn't even for the latter, it was only for staff.] If Slack would do something intelligent like let users expand threads in-line instead of forcing that infuriating two-column environment, that would be a big step forward. Discord's quote-reference mechanism lets you sort of "thread" things your own way and is far easier to understand. Grenadine's event-management backend decidedly has its issues, and takes a lot of getting used to. It's generally slow AF, but once you get a handle on how some of the filtering and display-column selection works, you can work through it without screaming. It's clear that parts of the UI were designed at different times by different people, and has a lot of inconsistencies. It didn't help that the Thursday before the con, as we were trying to hammer lots of event setup into it, they were also serving some big 2-day teachers' conference in Calgary out of the same database. Around 11am the entire thing fell on its face, and they had to scramble around standing up more database servers and eventually got things limping again. [Meanwhile, the Calgary conference threw up a public page with ALL their Zoom session links plainly on it, which was hilarious.] On the front end, the schedule presentation is a little weird and Peter wound up hacking around some of it with a little creative javascript work. It was still a little hard for con attendees to find stuff, and it wasn't easy for us to change the layouts and things to click. It took at least two steps to get from an item on the schedule to finally launch the Zoom for it, which I thought was pretty klunky, and that whole interaction was fraught with caching problems that reloading the page would NOT fix. It was another exciting time over at Helpdesk. For panel hosts and participants, there's a special page within each session that they have access to, giving the various launch and join links for the corresponding Zoom session. As role-oriented as Grenadine seems to be overall, these pages are not separated by role -- all of the panelist links *and the host start link* are displayed to all relevant viewers. So some of the time we had panelists coming in as hosts or co-hosts because they didn't see the link with their name on it, they just clicked the "big one". Nobody but assigned *hosts* should see that. This is a separate problem from panelists clicking the big bright blue tempting "start webinar" button after being made co-hosts within the Zoom, which was also a frequent issue. Eventually our hosts learned to *warn* them not to do that. Grenadine calls directly out to far too many CDNs and js-library repositories for my taste, and bringing a lot of that in-house and locally vetted would mitigate some of the obvious risks. I flat-out asked our Grenadine support exactly which of about a dozen third-party components I actually needed to permit to make the site work, and got a mostly correct list back. Many people seem to understand the problem, but fail to do anything to correct it unless they're working in critical PII environments like financial or medical. The per-session "chat" window that Grenadine has recently introduced was pretty much useless; the consistent "channel per event space" model that Arisia's Discord ran was far more intuitive and less socially isolating. At least the Grenadine site folks took away the hundreds of "<name> is entering" messages that were spewing through these as people accessed the session pages, leaving only what people actually typed in. The rest of this is about Zoom, particularly the API integration. My first brush with Zoom-via-API was during Arisia, but I'd say we got a lot closer to it with what we learned around Boskone. Several of us actually spent time groveling through Zoom's API documentation trying to figure out what they meant about some things. There are numerous parameters one can pass to the "create meeting" or "create webinar" calls, and evidently no way to bring in default values that match what we set in all those little blue switches in the regular website UI. The API layer is its own thing. The default of { "practice_session" : false } for webinars is what bit Arisia in the ass, causing anything that was a webinar to go "open to the whole audience" at startup instead of waiting for a host to trigger it. Given that an API cannot know what's going on in a session, if the panelists are ready yet, when the *intent* was to open up, etc ... that's a stupid default, since almost every webinar most of us have ever been part of has had the practice session enabled for good reason. It took a while for the Grenadine folks to straighten that out, but they did manage to change the calls to create webinars so that was 'true". About half the Boskone sessions were webinars so this was kind of important, and saved us some amount of hand-groveling through all the Zoom accounts to set things manually. We still needed some of that to fiddle recording settings, as some things were cloud-recorded and some weren't. In the Grenadine integration, we -- and by that I mean our crew *and* the Grenadine folks -- learned, kind of the hard way, that our strategy to have hosts launch meetings and webinars was doomed to fail without extra steps. The integration was able to create the entities just fine, with scheduled times, the session names, etc all set up properly. Part of this is returning a generic URL to join the session, setting up the special links that the panelists use to pop into a webinar during the practice-session, and a special piece called a "start_url". That latter bit is a longer URL that self-authenticates enough to start a session under a particluar account in the first place, without the operator needing to know any of the account details. And it EXPIRES in two hours after its creation. So our sandbox-testing was all going fine, we were launching demo sessions and having our hosts try the whole process soup-to-nuts, and then a while later all that simply stopped working. All the startup links were now going to a Zoom page saying "sign in to start this meeting". Clearly, trying to pre-create all the meetings/webinars for the whole weekend and launch then out of Grenadine in the expected fashion suddenly wasn't going to work, basically busting us back to needing to "hand launch" every one of them, and we were like "wtf good is this, it doesn't help?" This didn't bite Arisia because those sessions *were* created mere minutes before being launched. We might have been able to do that except that with so many more webinars, we wanted to get the special join email out to the panelists well in advance so they'd have it. Trying to use the "create meeting" Grenadine tool again wound up creating a whole *new* session entity at Zoom, named exactly the same with the same time and other parameters but a different meeting ID. After which we'd have no idea which session was going to properly launch, let alone that it was redundantly garbaging up Zoom's database. Fortunately, there was a solution that we managed to brainstorm up, get Zoom support to clarify timing limits on, and have the Grenadine people come up with a "script" to implement it. Kudos to Henry Balen for taking point on that for us. Alongside the API calls to create the sessions there is also a "get meeting" or "get webinar" call, which returns various details about a pre-configured one -- including the long "start_url" with an UPDATED expiry time. So simply "get"ting the block of details gives you another two hour window to start it, but you need to get that new URL into the hands of the hosts. Fortunately, Henry's hack script was able to directly populate that straight into the database that our hosts were going to launch from, and all we had to do was "kick" the next block of upcoming sessions a little in advance of their start time. So there was a little flurry of activity every 90 minutes to do that, and move a couple of other bits of updated data around for our participants to find. Again, it was nice to coordinate this over the tech-chat, a well-oiled machine once we got stuff into muscle memory. We were initially concerned that trying to do this too often was going to hit some daily limit in API calls, per a loose interpretation ofhttps://marketplace.zoom.us/docs/api-reference/zoom-api/meetings/meetingcreate but it turned out that the "get" calls qualify as a lightweight enough API that we weren't in danger of exceeding it. I hit Zoom's support-chat-thingie in an effort to get them to clarify this, which was its own little episode of frustration. Semi-snark: as soon as someone on the other end of that kind of interaction says, or even types, "May I know ..." you know exactly what you're dealing with. Having the API doc parroted at me verbatim without the real explanation I was looking for got old real fast. Given that Zoom appears to believe that the two-hour start expiry is a "security" thing, they're not likely to change it or make it configurable, so obviously this will have to become a running part of Grenadine and any other integration type that expects to schedule sessions farther in advance. Another security-theatre piece that caused a bunch of extra work was the stipulation that one must have at least one of a waiting-room or a passcode for entering meetings. This is Zoom's response to the flat-out pilot error common back in March/April, when people didn't know how to secure their own meetings. Grenadine by default was creating meetings with no passcode, presumably to make the joining process easier, but then it turned out that after go-time, hosts couldn't *disable* the waiting-room thus applied and would have to sit there paying attention and pounding the "admit all" button over and over. We wanted to make the hosts' jobs easier, but by the time we realized this was a problem most of the event particulars had already been set up. So we took *another* pass back through all the Zoom accounts with meetings and manually added passcodes. Fortunately, the "get meeting" API call would also fetch the new /j/MEETINGNUMBER?pwd=lots-of-gobbledygook format of join links and repopulate what attendees would later click on, so that worked out okay. It was hilarious that sometimes on adding a passcode to a meeting, Zoom would run into its own passcode complexity restrictions and not accept what *IT* had just generated. I had to fix quite a few of them by hand before the edit form would submit. Just because I haven't described it before, the process undergone when a browser hands off to the Zoom client is kind of interesting. For any type of link pointing to https://zoom.us/WHATEVER, a normal URL fetch is made up to Zoom, and what comes back is effectively a redirect to a MIME-type link in the form of "zoommtg://huge-amount-of-base64-after-it". If Zoom is properly installed on a machine, it knows how to launch the Zoom app and hand it that big block of muck as an argument, and hopefully the right thing then happens. On my Linux box, I get a pop-up letting me choose a different app to launch with, and selecting /bin/echo has some interesting results that also let me capture and debug a lot of the interaction. I was really glad I was in as part of this process and its discoveries early on, because this stuff could have put a serious ding in the con and had us all scrambling around doing "Plan B" and hand-launching sessions straight out of the Zoom accounts. With the Grenadine learning-curve and how we had to adapt things on that side, I'm not really sure which way would have been more or less work. _H* 210217 |