is this the next shiny migration?

For a few weeks, I've been working on migrating an app I built from Shiny to Next.js. The app is a dashboard for helping a state workforce agency share and interpret unemployment claims data internally. It was originally built several years ago using Tableau, and over the last year I worked on creating a Shiny version replicating this Tableau original. Now, I'm taking that transition a step further, and moving from Shiny to a JavaScript implementation with Next.

Infrastructure needs are driving this further update. I like using Shiny for dashboard development, but our tech team lacks experience in deploying Shiny apps. This means that we're relying significantly on vendor software for hosting services, which, as we began to think about scaling the total number of users on the app, has started translating to higher costs: both from the direct costs of additional licenses and the indirect costs of the DevOps folks learning to scale these third party tools. Perhaps even more importantly, this also means that a lot of our current deployment strategy relies on my ability to translate what I think Shiny does under the hood to an explanation that our tech team understands. This has not led to a smooth deployment process even with ten users, let alone the hundreds we are hoping to scale to.

Containerized Node.js applications, on the other hand, are already our tech team's go-to option for app deployment. They have established workflows and a deep understanding about the right way to securely deploy and scale applications built with these tools. This means that if I can get a version of the dashboard that runs using Node inside a Docker container, I can hand off the hard work of deployment to the tech team. So, that's been my goal. The first step, of course, was to actually rewrite the app.

Baseline: the Shiny app

The Shiny version of the app is built using the following R packages:

• Built with a package structure using golem
• Base UI built with bslib, with highcharter maps and charts and datatable tables
• Data queries handled with DBI and dbplyr from an Amazon Redshift server
• Global state held using the "stratégie du petit r"

The basic idea is that each of the three dashboard pages has its own golem module (just like a normal Shiny module, with its own UI and server functions). This function contains the core bslib components for the page layout, namely, card() components holding each piece of output. The pages are all stored as tabs in the highest-level app_ui() function using page_navbar(), to which I also pass the sidebar code holding a number of input devices (that is, selectInput() and sliderInput()).

Within each page, each output (graph, table, or map) also gets its own module: stored in, for example, mod_graph_trends_over_time.R. This extra layer of modularization keeps us from having to include the necessary wrapper functions for each piece of output (think RenderBlank and BlankOutput) in the same file as page layout. Other than those functions, these output modules are mostly empty - the actual meat of the creating each piece of output is offloaded in an additional processing script (mod_graph_trends_over_time_fct_make_trends.R) which handles the data processing necessary to prep the data and create each piece of output.¹

This is an interactive dashboard, so each output varies on input from the user. For example, there is a selectInput() in the sidebar to choose the week of data the outputs display. To ensure that all outputs are based off a synchronzied state, I:

• Store all of these input values in a reactive values list r, created in the main app_server() function.
• Pass r down to both the inputs and outputs in order to maintain a synchronized state.
• Filter the data according to the values stored in r within each output creation function.

This is called the stratégie du petit r. The aim is to create a "a small, internal database that is passed along through all the modules" used in my application, to make whatever inputs are necessary globally available without additional prop-drilling.

To this end, I also store the data itself (as a lazy evaluated call to the Redshift database) in r, which means that these inputs and outputs are directly available whenever and wherever the data is. There's a bit of additional hairsplitting I do with applying some filters directly to this lazy-evaluated data object before passing that call to any of the sub-modules, but the basic idea is still to handle most of the filtering within each module.

All combined, this set-up then leads to a relatively nice looking app which maintains a consistent global state both within and across pages. It's a little slower than I would like, given that each piece of output has to be computed and rendered sequentially on each page before anything is visible, but, otherwise, it works okay.

A Next.js version of the same thing

I found it surprisingly easy to recreate this setup as a Next.js app. I chose Next.js mainly because it's what our tech team prefers - for my personal projects, I've been just as happy with Remix. To carry out the transition, I used the following workflow:

• First, I recreated the main UI layout (the separate pages, sidebar, cards) using Tailwind and Shadcn. Here, I made each page its own route - i.e., /dashboard/page_1, and enclosed the corresponding cards in that route, with shared UI stored in a layout.js file for all of the child routes. It was nice to see the skeleton of the app in front of me from the very beginning, both visually with UI and mechanically with how these routes were structured. This meant there was a pre-existing home for each output component as soon as I created them.

• Next, I created React components containing draft versions of each output, based entirely on dummy data. To do this, I would first write up the fake JSON data containing the information necessary for the output - for exmaple, a hard-coded JSON file containing a random number representing the observations in each county in the state. Then, I would write the UI code to create the output based on this JSON. I used Highcharts for most pieces of output, and thankfully, because I used highcharter in the Shiny version of the app, there was a fair amount of configuration that I was directly able to copy over.

• After that, I ensured that the values from the input controls were accessible to each of these output components by setting up a global state store using Zustand. I was most surprised by how easy this step was - it required almost no configuration, and worked exactly how I wanted it to, serving as a pretty much exact stand-in for my Shiny app's "petit r" reactive values list.

• For each output component, I next created a corresponding API route (like /api/page_1/trends_data). I found starting with the fake JSON to be extremely helpful here, because I knew exactly what format the output of each endpoint needed to take. All I had to do to create the endpoints was write the processing code to get the raw data into that structure. This also meant another nice parallelism here with the R code, with these API endpoints serving as the partial counterpart to the data preparation functions (mod_graph_trends_over_time_fct_make_trends.R) in the Shiny app.

• Finally, I had to link up the UI with the data by actually inserting the API calls into the output components. Coming from Shiny, this part definitely felt the least natural to me. Because Shiny is handling the "API calls" (data refreshing) behind the scenes, you don't have to think about it so explicitly in that setting. Luckily, I used React Query here to manage this data invalidation - you pretty much tell it what variables change for the API call, and it handles the rest. I was again extremely impressed by how easy it was to set up and how well it worked once in place - especially with how well it handled caching. There was some weirdness with making sure that everything refreshed properly as the input values changed, but nothing that took more than a few minutes to diagnose and figure out.

Once I completed each of these steps, I had a decent first draft version of the Node dashboard up and running locally. From there, putting it into a Docker container and sharing it with our tech team was also straightforward. As of now, they're still working on setting up the actual deployment of it with a connection to the actual data and the necessary security precautions in place, but promising signs point to at least an MVP being available for user testing this week.

Takeaways from the switch

For me, I think the blessing and the curse of moving from building apps with Shiny to using Next.js (or another JavaScript framework) is that it puts you one layer closer to how your code is actually being run and interpreted by the browser. Working in Shiny, I've had many moments of frustration because the code is not being rendered in the way I expect and I can't figure out why. There is something going on under the hood with the reactive invalidation or the translation of the code to JavaScript that is 1) not immediately apparent and 2) different from what I would expect. This leads to apps that work well when they work, but which are hell to debug when they don't.

Moving from this context to working on a Next.js app feels like getting to look inside the engine of the car. Rather than working with a series of wrappers, there's a sense of working from first principles, where you can directly see how each line of code is appearing and altering the app. You suddenly have a lot more control over both the data processing (like through query invalidation) and the UI (by building your own React components). This control is incredibly powerful, but it's also scary. It requires at least a partial understanding of how the client and server interact through the browser, an understanding that many academic or data scientist-y type folks won't have the desire or time to develop. Shiny gives you a shortcut around needing to fully develop this understanding, and because of that I still think it's an apt tool for many use cases.

On the other hand, if you find yourself bumping up against the limits of what feels capable in Shiny, then getting more in the weeds with Node and React might help you kick down those walls. There is a reason that these tools are used for so much of modern web development! And while these tools might not normally be the standard kit for dashboards and data analysis, I did find that all of my Shiny experience made it much easier to plan the structure of my Next app and understand the challenges that I faced. The Venn diagram of the skills used here is not two separate circles; learning feeds into learning, and I've felt myself move closer and closer to seeing around the syntax and understanding the mechanisms at play underneath.

It's lots of fun!