Dan Lu

Recurse Center Halftime Reflection

Sun, 02 Nov 2025 00:00:00 +0000

I’m halfway through my 12-week batch The Recurse Center (RC) and thought I’d at least reflect on what it’s actually been like. Having graduated from NYU this past May and remained in the city, I’ve been fortunate enough to get accepted into RC this past September and attend in-person at the RC Hub.

tl;dr: My RC batch experience has felt like a multi-swimlane “choose your own adventure” hackathon of sorts, where the projects, people you work with, and the durations are entirely up to you. Because of the self-directed learning environment and community of nerdy and generous people at RC, this becomes a very energizing place for me to learn all sorts of programming.

Three quick things before getting into it:

In the style of The Timeless Way of Building by Christopher Alexander, I italicize the parts that describe the gist of what I want to convey
- Unformatted text provide additional details for curious readers, but feel free to skip :)
These views are strictly my own, at this particular moment in time. They don’t represent the views of RC members and faculty, and even more so, probably not even myself in the near future after the next six weeks of my batch. Still, I hope it gives others a taste of the RC experience.
Unlike most things I write, I’ve decided to include the background section at the end of this post.
- It’s an optional appendix-y thing to read. It covers the lead up to RC after I graduated from NYU with a PhD. The main purpose of this blog post is to reflect on the RC experience itself and less on what I did before that.

What I expected RC to be like

RC labels itself as a “writer’s retreat but for programmers,” yet I am unfamiliar with the event model of a retreat. Before RC, I thought it would be a sort of “architecture student studio” but for programmers to explore their own coding projects and exchange their ideas formally or informally within a shared workspace.

In architecture schools, students generally share a large studio workspace. Even if they are in the same design class, eventually, everyone ends up working on their own design projects exploring their own ideas. Imagine the desks slowly getting filled with wildly different drawings, cardboard, and foam models of various shapes and scales. Yet, even if each student’s ideas are quite different, the nature of the shared studio space is such that eventually informal conversations between students could still end up being very informative and educational discussions. Architecture schools refer to this as a “studio culture” and point to it as a fundamental aspect of architecture education.

Since my college days, I have been to various makerspaces at NYC in search of this kind of studio culture vibe but for broader “designer/maker/tech” folks who want a space to work on their projects. I had also went through a one-month creative tech + arts “unschool” program called ITP camp. To me it was the closest thing to the studio culture, though way less formalized and more free-form. RC also referenced its influence by the “unschool” movement, so I initially felt confident understanding it that way.

People at ITP come from various sides of the art-tech spectrum and give workshops to teach and share what they know. All the while, people work on whatever projects interest them and showcase them at the end of the month.

So overall, going into RC, I figured my batch experience was going to be some cross between ITP camp and architecture student studio but focused very much on computer programming.

Little did I know…

What RC ended up being like

…I was kinda right! (haha, no real plot twists here)

I’d say that in a broad-strokes high-level sense, RC has been like ITP camp but with greater emphasis on programming instead of creative tech and arts. But there are so many “smaller” details that add up to a significant difference between what I expected and eventually experienced. I’m honestly still figuring out what those differences are, but I’ll say for now that RC seems to have reduced a lot of friction points that normally occur in other learning environments.

This is not to say that RC is better than ITP camp or other programs out there, because that ultimately depends on your goals and priorities w.r.t your career and what do you want to learn. This is also not to say that there are no “creative tech people” at RC, since creative coding is a big part of RC.

Overall, I am grateful to be in such a welcoming place where I can share my unfettered excitement about programming topics, while I also develop my ability to direct that excitement effectively towards helping others learn toward their own interests and ideas. I ultimately see RC as a great place to figure out:

What do I actually want to learn?
How do I best help others learn what they want to learn?

Having been in various roles in academic research, general industry, makerspaces, and unschool-like programs, I have found RC to be very unique in its ability to really encourage me to focus on those two questions towards learning programming.

Let’s start with something similar between RC and ITP camp: learning is very much self-directed. No one explicitly tells you what to work on here. That’s ultimately up to you.

Weekly meetups and workshops are the most structured ways of learning at RC. But again it’s up to you whether you want to keep going to them or how much work you want to put into them. There’s no course add/drop deadlines, no exams or quizzes, or even homework.

Sure, certain groups will involve reading books or doing coding exercises before then but you are always welcome to join and participate however you can, even if you don’t always do them (and no one holds it against you for “dropping out”).

One thing that’s really helped so far are the “volitional muscles” workshops organized by RC faculty. Here you mainly self-reflect on key questions like “what do you wish you could do w/o constraints” and “why do you want to do XYZ” and then discuss with others in the workshop…and that’s all! No one there tells you what the immutable rules to life are…because you have to decide for yourself what you value in your projects and how to focus more on that.

I spent the first week attending pretty much every meetup/workshop event then slowly narrowed it down to a few that were of particular interest to me. Some have been structured around book chapters or coding challenges that people try to complete beforehand, while others are more free-form in sharing ideas or working on the day of. Regardless, it’s usually the case that at some point someone will share something that prompts me to take another look at what I think I know already:

In the Building LLMs from Scratch book group, someone simply asked why a formula had 1/sqrt(d) instead of just 1/d…and that led me to dig deeper into machine learning theory and how to normalize values such that neural networks are unaffected by the number of embedding dimensions.
In the Machine Learning/Reinforcement Learning for Games group, some folks have demonstrated that heuristic models instead of deep neural nets can still do surprisingly well for certain tasks. This is leading me to continue thinking about when more sophisticated methods are actually superior and what games or benchmarking metrics are most appropriate at identifying a method’s performance.

And then there’s Zulip, the online chatroom app for Recursers (in-batch and alumni) to discuss whatever we find interesting. Yes, “it’s just like Slack/Discord”…except in the context of RC it feels totally different. One big difference is the “checkins” channel. Each person gets their own “thread” and can say as much or as little as they want about what they’re working on. This ends up feeling like everyone has their own micro-blog and conversations naturally form within each thread when a person’s project touches on other people’s interests a lot. The net-effect is that you feel like a part of a deeply motivated community of learners.

I personally try to write out some TODOs at the start of each day and get motivated even when people just respond with emojis. Just clicking through other people’s check-ins is enough to keep me pumped and motivated to go back to work on my own thing (or, better yet, distracted into another topic that’s been on my mind anyway)

The sheer volume of information exchanged in the other channels and even direct messages has sometimes left me in awe…which re-invigorates me to keep learning and coding with the same amount of passion.

Weekly presentations are on a volunteer sign up basis and each person gets only 5 minutes. Coming from 10-20min academic conference presentation world, this was initially very challenging for me as a presenter. As an audience member, however, it’s a sheer joy to watch, get a little confused, see something exciting, or all of the above. I don’t fully understand every technical thing everyone is presenting, but there’s always something absolutely fascinating in there!

Folks at RC reading this will likely notice that I have not mentioned that much about pair programming, which I do think is a vital part of the RC experience. I have yet to pair program a lot on my projects yet but I want to soon.

Most recently, I did “pair” with someone over a simpler task: finding the right Python library for something. The two of us only spent about 10 minutes, but for me it was quite valuable because I imagine I would have otherwise been caught in an endless loop of perfectionism and confusion picking out what library to use. Instead, talking through my thoughts with someone else was super helpful in converging on something that seemed good enough and get started on actually coding (note: we also had the shared goal of using that Python library for the ML/RL Games workshop).

I did however try to reach out to as many people in batch my first week, even if I felt there was little chance we would actually work on something together. That meant both being open to various programming topics but also not feeling too bummed when certain people expressed moving onto something else.

I’ve also had one-on-one meetings with RC “Faculty.” I put quotes on that word because I come from academia and find that my conversations with them are very different in nature from meetings with university professors.

Here, the difference between RC and other unschools becomes especially clear because RC does have a dedicated career services team to help you potentially find a job. At the same time, it doesn’t insert or interfere with everything else at RC so that you can stay relatively focused on learning programming. It also helps, in my opinion, to not see RC Faculty as purely a career services team like those found in universities.

Rather, I am starting to appreciate how RC Faculty are there to make sure you can get the most out of the batch experience, whether that means learning to code or finding a job.

One of them has kindly told me that they “are not my PhD advisor,” which wonderfully brought to my attention how I may had been acting in the preceding conversation listing my “progress items” so far. To be clear RC does not put ironic quotes around Faculty, with the best of intentions.

This last point actually brings me to emphasize something important that I wish to continue processing and understand better. RC may in fact be so different from other learning environments than what I’m used to such that:

I will need to be more attentive to how I contribute to RC, especially if I am not as generous as I could be or taking things too seriously or personally.
And I should still be kind to myself about this process, that it’s OK if I struggle with the above, because this environment is still very new to me.

A related thing is this apparent yet wonderful paradox I’ve noticed between people sharing something they are deeply and personally interested in while also not taking things too personally when listening to feedback and comments from others. Thanks in part to the social rules and self-directives at RC, I think there’s a shared trust among Recursers that anything we say are some attempts at thinking out loud while still being generous towards each others’ passion for learning. We might still end up breaking the social rules, but we ultimately feel like we are all in this together.

The social learning experience of RC is something I never could have imagined. I have to say that I’ve been frequently caught off guard by how often I felt touched, inspired, and really seen at RC.

Even when I just want to take a “quick break” from a solo coding project, I sometimes find myself walking around the RC Hub space and getting pulled into something else. One time I passed by a conference room where folks were trying to solve a reinforcement learning problem in the Sutton-Barto textbook and I got ushered in. Soon after, we ended up on a tangential conversation about artificial agents and how they might demonstrate language as an emergent phenomena in a simulation that only required their survival.

In short, RC has felt more like a multi-swimlane “choose your own adventure” hackathon of sorts, where the projects, people you work with, and the hackathon durations are entirely up to you and others interested in a particular topic. If that all still sounds vague, it’s because that is the nature of RC’s relatively unstructured learning environment, where the real value comes from the interactions you have with other Recursers.

Fun Things I’ve Learned

Here’s just another lens to view my experience so far - all the random things I learned from other Recursers, both programming-related and not programming-related at all:

Paradoxes in programming are definitely a thing.
In this age of LLMs, don’t forget about simpler natural language machine learning tools like Natural Language Tool Kit (NLTK).
RAG agents can also work with graphs now.
You can do a lot of Photoshop-like things using CSS and SVG masking.
I should probably structure my machine learning Github repos better, especially in lieu of the Lightning-Hydra Template.
Functional programming languages like Elixir can also be used to play around with linear and non-linear functions.
Cycle detection for recognizing when program outputs are repeating can be done via the cute-sounding tortoise and hare algorithm.
Memory in human brain has been found to rely on distributed representation, as somewhat illustrated in the movie Eternal Sunshine of the Spotless Mind.
Reinforcement theory is a perspective on understanding human motivation and something I hope to tie to reinforcement learning.
Tu Youyou won the Nobel Prize in Medicine for her work in applying traditional Chinese medicine to effectively treat malaria.

I have no idea how a lot of this can be applied to my future career or personal projects in programming, but I feel such joy just knowing about these things as seeds for future explorations.

Things to keep in mind moving forward

So what’s next? To be honest, I am still working out the things I want to work on in my next six weeks. For now, I’d like to at least distill some vague-yet-important principles on how to approach the rest of my batch:

Be kind to yourself about how you interact with others so that you can learn more generously with others.

Try to pair program more. Maybe that doesn’t mean “finding a whole new project” that others are interested in, but noticing a particular task within a project that would be interesting for anyone to even just look at with fresh eyes.

As energized as I get from being in the zone that is RC, I should try to also tend to other parts of my life (personal health, relationships, etc.). As one of my favorite game developers has said, those “other parts” should really be at the core of my identity as a programmer.

No one here gives you definitive answers. Instead, appreciate how:

people will offer suggestions
people will raise more questions
you get to decide on those answers yourself

When in doubt, take time to collect your thoughts and recover to a place of peace or stillness…then just code something!

Background

After graduating from NYU with a PhD, I thought at first I could find a job on my own while doing self-guided online courses and projects that would: 1) showcase some data science and analyst skills that I feel are missing from my current resume, and 2) involve some ideas of interest to me that I didn’t have time to really explore during my PhD.

I may go into details about that experience another time, but suffice to say that the first few months after graduation were not pleasant. It was quite a grind just to work consistently by myself, whether that meant working on SQL Leetcode, taking online courses that got boring after a while, or feeling stuck inside an initially interesting yet job-skill focused project that quickly turned into a self-created hell.

Around this time, a professor/colleague of mine from NYU had shared RC’s blog post on how AI has affected Recursers’ experience in coding and, in particular, learning to code. This was not my first time hearing about RC. I had applied to RC before my PhD and didn’t get accepted. My disappointment from that experience since then had left me thinking it would’ve been difficult to apply again and get in.

However, reading that RC blog post led to me poking around RC’s User Manual and other pages. To my surprise, the various themes and ideas, especially the self directives, started to really resonate with me, unlike the times I read them before my PhD when I applied the first time. Reading about ideas around volitional muscles for approaching coding projects out of genuine interest rather than fear around getting a job even prompted me to reflect on my last couple months struggling to build projects to counteract perceived skill gaps in my resume (notice my prioritization in the first paragraph of this section).

More broadly, it made me recognize the valuable moments of my PhD where I had to exercise those volitional muscles to decide which parts of a project are most important, and it prompted me to re-evaluate what I expect out of RC in relation to the job search (i.e., nothing can really make up for lack of job experience until you get a job, so what’s the point of coding personal projects?)

While I was initially interested in RC as just a place to work on my own projects, this “second reading” prompted two thoughts:

Apply! I figured at the very least I could get a conversational interview, which meant at least a chance to talk through some of the self-directives when it came time for questions from me.
I also figured that three things had changed since I first applied to RC:
- I had a stronger direct passion for programming, ranging from machine learning to creative coding.
- I had gained a lot of coding skills from my PhD.
- I felt like I had a very different mindset about project-based learning.
Even if I don’t get in, I’ve learned to approach my next few months of job search and coding projects with a newfound sense of joy, patience, and motivation.

Two interviews with RC later and I was fortunately accepted.

Minika

Wed, 01 May 2024 20:52:00 +0000

A minimalist remake of that watermelon game in Unity.

Code is on Github: https://github.com/holistudio/minika

AECTech Hackathon Takeaways

Sat, 04 Nov 2023 00:00:00 +0000

I had a lot of fun this past weekend at the AEC Tech Hackathon at NYC. I really appreciated the open format of the hackathon. The whole event started with an “open mic” - anyone could walk up to the microphone at the front of a large room and pitch an idea they wanted to work on in the next 24 hours. Teams would organically form around the big ideas that more people were interested in. At the end, the presentations showed a wide range of ways of using AI in AEC applications.

Yes hackathons are a great place to share solutions. But they seem like an even better place to share problems. So I came away from the event with a lot of takeaways regarding both the promises and challenges of AI in the architecture, engineering, and construction industry.

Boring is the new sexy: From the open mic, it was clear a lot of people just wanted an LLM or app to do something very time consuming and boring - rename all the rooms in a Revit model, review submittals, or help track progress on jobsite. The challenges didn’t sound very “sexy” or “cool” but addressing them could have profound impact on people’s productivity. To paraphrase Mostapha Roudsari, “Automating names of rooms is boring, but automating that inside Revit is exciting!”

Generative AI renders all you need: Generative image models like DALL-E and Midjourney were interpreted by media outlets as “creative AI” - something that could augment the creative process of artists, creators, and designers. Some architects I spoke to admitted they were more interested in giving those generative image models a screenshot of their grayscale concept 3D model to output realistic 3D renders. Effectively, they saw generative image AI as just a time-saving replacement for the 3D rendering engine they typically use to show realistic images to clients.

Is AEC data AI-ready?: Now that machine learning models have demonstrated capabilities on more general purposes (think YOLO for object recognition), now those in AEC tech want to re-purpose them for specific AEC needs. I met a software engineer in a window manufacturing company who wanted to use AI to help catalogue their shop drawings. I saw this as a matter of finding an image classification model like YOLO and fine tune it via transfer learning on labeled CAD drawings…and that would’ve taken more than 24 hours for sure. Those in Big Tech envision that other businesses can one day buy a general “off-the-shelf” AI model that can be fine tuned for specific industries. Until that day arrives, those in the AEC industry may still need to take a hard look at the data they have and decide whether it’s “AI-ready.” Taking the window manufacturing company’s CAD drawings as an example: are the parts of the drawing clearly labeled/annotated? Will the AI misinterpret those labels? Are any of the parts mislabeled? AI developers will have to figure out how “off-the-shelf” AI model can catch human mistakes in the dataset, and AEC developers will have to figure out if their datasets are of sufficient quality.

A Universal Assistant but for AEC: A similar desire seemed to be expressed in a lot of projects that tried to control ChatGPT for automating BIM workflows. One project called itself Clippy and notably kept prompting ChatGPT until it’s coding solution actually worked on Revit. So while AEC folks want an AI tailored specifically to the AEC industry, they also seem to want a general purpose AI to be a “really smart AEC intern” that helps out with anything they come across in their workday. So in line with that future vision of an AEC company purchasing AI off the shelf, AEC folks need a clear understanding of what they really need from that AI and clarify what efforts are needed to re-train and specialize that AI for AEC needs.

AEC + XR = TBD: There seemed to be only one VR project in the whole hackathon. I don’t think this means VR is dead, but the AEC industry’s enthusiasm for VR seems a bit dormant right now. My bold prediction remains that once Embodied AI takes off, it’ll be really fun to interact with AI in VR while designing built environments. Right now the AEC industry is still not sure how useful the Apple Vision Pro will be, even if AEC tech folks totally get the idea of spatial computing. Something to keep an eye out in future AEC hackathons!

It also helped that I worked on a project, FURN-E, that won the Best Open Source Hack category of the hackathon!

As an architect explores design options with images from generative AI like DALL-E, FURN-E can detect furniture objects in those images and automatically suggest links to similar real-world products for procurement from manufacturers in subsequent construction phases of the project.

Knowing this kind of tool can be repurposed in many other ways by designers looking to make generative AI images into a physical reality, we have made this project available on Github

Hilma af Klint: Engaging and Transcending Scientific Discovery

Tue, 04 Jul 2023 00:00:00 +0000

I went to the Tate Modern the first day I traveled to London. I got there in the afternoon, right before it rained, walking over the Millenium Bridge.

I was quite exhausted. First stop, the espresso bar.

The main thing I wanted to see was an exhibit on Piet Mondrian* and Hilma af Klint. I honestly went because I recognized Mondrian’s name from the one architectural history class I took in college. Had no idea who Hilma af Klint was.

This was the first thing I saw in the exhibit:

Maybe I was tired. Maybe I’m just dumb. But I really thought the quote was intentionally shown with parts of it missing. And then I walked closer to the wall. Turns out, the quote is fully visible up close:

And that’s probably when the espresso kicked in.

All of the sudden, I realized it was the way the text paint reflected some of the gallery lighting, causing the text to appear like parts were missing from afar. What was visible to you depended on where you stood relative to the wall. And just as the quote suggests, I became self-aware of the nature of my own perception. I was in awe.

I asked a museum staff whether this was intentional and they weren’t sure. I’d like to think this was an intended feature, because this interplay between perception, nature, and understanding was a consistent theme in af Klint’s works on display.

And walking through the rest of the exhibit, I began to learn more about Hilma af Klint and the world in which she worked as an artist. One of the exhibition’s main themes is how af Klint’s artistic process was deeply rooted in the scientific and religious discussions of the time. So artifacts and drawings by Carl Linnaeus and Rudolf Steiner** were placed side by side with af Klint’s sketches, revealing connections between contemporary ideas and her own process.

Left, flower classification diagrams by Carl Linnaeus. Right, diagram sketches by Hilma af Klint.

Though a lot of Rudolf Steiner’s ideas are considered religious pseudoscience today**, I do recognize that he was trying to coherently frame a more unified theory of human existence to come to terms with both spirtual ideology and the latest findings by contemporary scientists that resolved some of life’s mysteries, the unobservable and the observed.

Left, chalkboard diagrams by Rudold Steiner. Right, works by Hilma af Klint.

Even today, I think people who know they should maintain an objective and scientific mindset still turn to religion and spiritual ideas because they desire some sense of meaning in their lives, a connection to something greater than their daily to-do’s. And so in the context of Steiner’s anthroposophy (or “spiritual science” as Steiner called it), I see in af Klint’s work the constant attempts to propose narratives that elegantly tie together the mystical spirit of life and the empirical models of the natural world around us.

Though I didn’t really care for the Christian symbolism in her work, I found myself relating to af Klint’s intent and process in so many ways. I’ve always been intrigued with how people still find meaning and beauty in a more scientific and technologically driven world like Feynman’s description of a flower, and the entirety of Zen and Art of Motorcycle Maintenance. I enjoy coding graphics with p5js and the broader pursuit of using systems of logic and patterns to make art.

Hilma af Klint - Group IX / SUW, no. 1 and 8. The Swan 1915

But most of all, I really wonder what kind of art would af Klint make today, as ChatGPT and Midjourney generative AI has captured the fascination and concerns of artists and the general public. af Klint and Mondrian may well be concerned about the ethics of reusing art, including their own works being fed into training generative image models.

I would like to think that they would also look past generative AI as a tool for remixing creative works and be immensely curious about the machine learning research that went into building those generative models in the first place. My impression from the likes of Geoffrey Hinton, Yann LeCun, and Yoshua Bengio is that fundamental machine learning research does build off an understanding of how humans learn about the world around them. And so I’d bet that af Klint would eagerly sit through a lecture on the history of machine learning and recognize neural networks as major abstractions that distill parts of how people perceive reality.

Some of her quotes and diagrams did suggest an interest in iridology, the study of patterns and colours on the iris of the human eye. I can’t imagine how af Klint would re-purpose her own skills of abstraction and artistry to really engage with the theories and methods of computer vision and deep learning and their sources of inspiration in cognitive science, linguistics, and human anatomy.

Iris Diagnosis Charts by af Klint

I wonder what af Klint would think of convolutional neural networks today, as they essentially look past an image’s raw form of pixels and try to associate relationships between that form’s features and verbal descriptions.

But since I can’t imagine how af Klint would respond to the fascinating theories behind machine learning, I’m ever more inspired to make something of my own in her spirit. Further encouraging me are the various genres her thoughts seem to span, beyond painting and drawing. The exhibit displayed notebook sketches with sheet music sketches and architectural models.

Left, digital scans of af Klint’s notebook which contains sheet music sketches paired with other diagrams. Right, museum model makers’ interpretation of a temple which af Klint sketched and imagined where she wished people could visit to experience her work.

I left the exhibit with a strong sense of how art can take inspiration from emerging science and technology. af Klint really danced with the ideas of her time and produced works that prompted viewers to look past their physical reality and find something transcending that existence. While I don’t think I would agree with the entirety of af Klint’s thoughts word for word, I did feel a strong connection to her process and approach to art and storytelling.

“I am an atom in the universe that has access to infinite possibilities of development. These possibilities I want, gradually, to reveal”
-Hilma af Klint, 1916

I should make sure to drink espresso before every museum/gallery I visit…

___

*So about Piet Mondrian…

There were a few paintings of his I liked here and there. And the exhibit did seem to hint at how, like af Klint, Mondrian was also inspired by science and theological discussions of the time. But honestly those works did not really leave as much of an impression on me.

The few things that stood out to me by Mondrian:

He had a theater design piece! This was surprising at the time I visited, since I only remember Mondrian as a painter (of course now I remember Mondrian was a member of the de Stijl movement)

A lot of his works employ visual patterns that might make fun Processing coding exercises.

**Carl Linnaeus would also attempt to classify human beings into hierarchies or “varieties” based on geography and skin color, thus contributing to the eugenic ideology. Rudolph Steiner’s theosophy found some admiration among future Nazi party members. One should always be cautious in embracing ideas to their fullest extent and question if their application to other segments of society overrides important moral and human values.

Project Spaceship

Wed, 14 Jun 2023 18:47:00 +0000

What are we trying to do?

Imagine you start stacking a few Lego blocks and then an robotic arm* starts stacking blocks alongside you. Ideally, the robot’s AI can intuitively pick up on what you are trying to build and suggest blocks towards your goal. Or even better, the AI’s blocks inspire you to take your build into a different direction you never imagined beforehand!

What would it take to get such a human-AI co-creation system to work?

How would we train the AI to to build different shapes, block-by-block step-by-step?
How would we train the AI to recognize what shapes are being built, given only a few of blocks placed in front of it?

And on the other hand, how would it feel for human beings to interact with such an AI?

Could we build with the AI without any verbal communication or text prompts?
Would we really appreciate the blocks placed by the AI or get annoyed?
Could we get inspired into new directions for making something, or just converge with the AI on a single shape?
(Or will we let the AI take over since it seems so much more intelligent than us?)

What is new about our approach?

Large language models, like ChatGPT, point to new ways of verbally communicating with AI. Generative AI, like Midjourney, point to new ways for designers to iterate and curate design options with text prompts.

But designers operate beyond the space of natural language. Design can be a language all on its own.

Current research on embodied AI focuses on related issues. How can AI develop a higher level reasoning and learn through multi-modal interactions with the world?

An embodied AI that can understand designers’ implicit intents and directly manipulate design models’ components may present a new kind of user experience with its own benefits and challenges. But we won’t know what those benefits and challenges are until we really get there…

If we succeed, what difference do we think it will make?

For now, I’ll start by training an AI to build and identify shapes on ShapeNet. Then I’ll make a Unity3D game where the user stacks blocks to make a common 3D shape and the AI can then stack a few more blocks. Hopefully the AI can correctly identify the shape being built and assists the user in completing that shape. This project will serve as a “toy data experiment” for an embodied AI co-creating with a human-in-the-loop.

But the same approach can one day lead to various AI co-creation applications for designers of all sorts: architects during concept design, industrial designers developing manufacturing sequences…or a kid just making their own spaceship!

Excited?

This project is currently in progress on my GitHub.

Check out my current progress on my dev log for this project here.

Footnotes

*This can happen in either the real-world with a robot arm stacking blocks OR a virtual/extended reality environment with 3D model blocks appearing wherever you/AI wants. For now, getting something working in a 3D game environment (i.e., Unity 3D desktop) is a good place to start.

Hello World!

Wed, 14 Jun 2023 00:00:00 +0000

Hello world!

After many years of learning computer programming, I finally decided to make my own personal webpage outside of Squarespace. Don’t get me wrong, Squarespace has its usefulness - like for me when I was still learning web frameworks but needed to show a design portfolio of projects to potential grad school programs and employers.

Lately though, I felt that it was a bit silly to keep paying that subscription fee while I am now a pretty confident programmer of sorts. Having ChatGPT helps too (though it took me a while before I realized it was suggesting code using Jekyll version 4.x!).

Here’s to a more fun (and free) time finangling with Jekyll on a Github page!

Smart SafetyWear

Fri, 05 May 2023 18:47:00 +0000

While maintaining public streets and highways, roadway construction workers have to balance their attention to completing their tasks with their overall safety in relation to the surrounding traffic vehicles moving around their work zone.

What if a wearable device (e.g., smartwatch) could intelligently warn workers of dangerous vehicles (e.g., speeding cars)? What aspects of worker behavior (head movement vs body movement) would be important to keep track of? What kinds of alarms (e.g., sound vs haptic/vibration) are effective? How could an AI, aware of a worker’s current behavior and surrounding traffic, pick the best alarm to raise on the wearable device? And how can we safely design and test such a smart wearable device without risking workers’ lives?

My PhD dissertation investigates how virtual reality (VR), wearables, and machine learning can be used to improve roadway worker safety the following ways:

1. Understand worker behavior: Collect data on how construction workers move during roadwork and train AI models to predict their movements (i.e., trajectory prediction).

2. Understand worker reactions to alarms: Collect data on how construction workers react to different alarms triggered by dangerous vehicle and analyze which alarm attributes (duration, repetitions) have the greatest safety impact.

3. Develop AI-based alarms: Use that collected data to train AI (i.e., reinforcement learning) to pick alarms that are most likely to prompt an individual worker to react safely and consistently over their entire workday.

All of the above methods can use a safe yet immersive work zone simulation where researchers can closely monitor a construction worker’s risks and behaviors in otherwise life-threatening traffic construction scenarios.

Towards a future where smart wearable devices can help prevent construction worker-related traffic accidents, this research demonstrates how VR provides a testbed staging area for prototyping novel AI applications prior to their real-world testing and deployment.

More info upon request by contacting dbl299 [at] nyu [dot] edu.

Technical Skills

Unity, HTC Vive API, Virtual Reality Systems, C#, Python, PyTorch, Reinforcement Learning, watchOS, RaspberryPi, Ultrasonic Sensors, Wearable Prototyping

Publications

Lu, D. B., & Ergan, S. (2026). Behavioral modelling of roadway construction workers: Improving deep learning-based trajectory prediction with contextual information in traffic work zones. Advanced Engineering Informatics, 71, 104277. doi:10.1016/j.aei.2025.104277

Lu, D. B., & Ergan, S. (2025). Principal attributes of wearable warning alarms to promote roadway worker safety. Advanced Engineering Informatics, 67, 103481. doi:10.1016/j.aei.2025.103481

Lu, D., Ergan, S., & Ozbay, K. (2025). Reinforcement learning-based optimal control of wearable alarms for consistent roadway workers’ reactions to traffic hazards. Journal of Transportation Safety & Security, 1-25. doi:10.1080/19439962.2024.2449119

Lordianto, B., Lu, D., Ho, W., & Ergan, S. (2024). Hapti-met: A construction helmet with directional haptic feedback for roadway worker safety. In: Riveiro, B., Arias, P. (eds) 2024 EG-ICE International Conference on Intelligent Computing in Engineering, Vigo, Spain, July, 2024. PDF

Lu, D., & Ergan, S. (2023). Predicting roadway workers’ safety behaviour in short-term work zones. In: Broy, T., Li, H., Lu, Q. (eds) 2023 EG-ICE International Conference on Intelligent Computing in Engineering. PDF

Qin, J., Lu, D., & Ergan, S. (2023). Towards increased situational awareness at unstructured work zones: Analysis of worker behavioral data captured in VR-based micro traffic simulations. In: Skatulla, S., Beushausen, H. (eds) International Conference on Computing in Civil and Building Engineering (pp. 63-77). Cham: Springer Nature Switzerland. doi:10.1007/978-3-031-32515-1_6

Responsive Outpatient Spaces

Sat, 05 Mar 2022 18:47:00 +0000

Recent technological trends in healthcare, such as telemedicine, wearables, and AI-augmented tools, are changing what patients expect out of their doctor’s visit and changing how healthcare workers expect to work in outpatient practices. Architects and engineers need to rethink their process of designing outpatient clinics’ physical spaces and building systems to be more flexible and adaptive to these changing expectations.

Responsive environment technologies (e.g., robotic furniture, smart lighting, etc.) can provide the means to improve the flexibility of interior spaces. Though these technologies have found some initial success in residential buildings, such innovations have yet to be applied to the different kinds of rooms in outpatient clinics.

Initial investigations into such applications have developed virtual reality (VR) models of robotic furniture and wall transformations: how a physician’s office can transform into a patient room (above left), how workspaces can transform into a meeting room between physicians and their assistants (above right). Healthcare workers interacted with these VR models transforming in real-time. While they mostly expressed an enthusiasm for these ideas, some healthcare workers expressed concerns for how the room transformations would fit into their typical medical workflows.

Future work should specifically identify the critical needs of outpatient clinic occupants, be they healthcare workers or patients. Those needs would then need to be mapped to appropriate responsive environment technologies that adapt to those needs at the right time and physical scales. By considering upfront how responsive environment technologies can serve building occupants, architects and engineers can create spaces that enable outpatient practices to serve their patients with a quality healthcare experience.

Technical Skills

Unity, Rhino

Publications

Lu, D., Ergan, S., Mann, D., & Lawrence, K. (2022). The Need for Responsive Environments: Bringing Flexibility to Clinic Spaces. In Construction Research Congress 2022 (pp. 812-821). https://doi.org/10.1061/9780784483961.085

Lu, D. B., Ergan, S., Mann, D., & Lawrence, K. (2020). A Vision for Evaluations of Responsive Environments in Future Medical Facilities. In ISARC Proceedings of the International Symposium on Automation and Robotics in Construction (Vol. 37, pp. 805-812). https://doi.org/10.22260/ISARC2020/0111

EASEEbot

Mon, 05 Apr 2021 18:47:00 +0000

In the United States, buildings account for 28% of total primary energy usage and 40% of greenhouse gas (GHG) emissions. Studies by the EPA and other agencies estimate that energy costs could be reduced by up to 20% by better air-sealing the building envelope (i.e., walls, windows, roof). While energy audits and envelope inspections are necessary to identify these air leaks and maintain a building’s energy efficiency, such undertakings can be expensive, time-consuming, and physically dangerous to inspectors.

Towards enhancing inspectors capabilities to assess building envelope conditions, we propose a robotic envelope assessment system for energy efficiency…or EASEEbot for short. The EASEEbot system has two use cases for human-robot collaboration between a building inspector and sensor-equipped robot: FLY and CLIMB.

FLY: The drone can navigate around the building exterior with a thermal infrared camera, simultaneously capturing a 3D point cloud of the building’s geometry and a heat map identifying areas of building envelope leaks. It can also navigate inside physically tight spaces, like roof attics, and perform similar camera surveys.

This can save inspectors significant time and physical labor, especially in both urban high-rises and low-rise residential buildings.

CLIMB: The drone can also be tethered and move along the vertical face of the building walls to scan the building envelope’s internal layers for moisture using a ground-penetrating radar (GPR) sensor (below right).

This can save inspectors the trouble of opening up sections of the wall for moisture damage evaluation, since the sensors can assess moisture conditions from the outside (i.e., nondestructive testing). This can also save property owners the setup costs for installing otherwise required scaffolds around the building envelope.

We proposed the EASEEbot system to the US Department of Energy in its Envelope Retrofit Opportunities for Building Optimization Technologies (E-ROBOT) Competition and ended up as one of ten finalists.

Since the E-ROBOT competition, the team has applied for a patent and spun off as a building inspection startup.

Team

Concept Design and Use Cases: Bilal Sher, Daniel Lu, Beyza Kiper, Semiha Ergan
Robot Hardware: Bilal Sher, Xuchu Xu, Chen Feng

Awards

Finalist in US Department of Energy (DOE) Envelope Retrofit Opportunities for Building Optimization Technologies (E-ROBOT) Competition Phase 1 (April 2022)

Publications

Xu, X., Lu, D., Sher, B., Kim, S., Rathod, A., Ergan, S., Feng, C. (2021). Conceptual Design of a Robotic Building Envelope Assessment System for Energy Efficiency. In: Feng, C., Linner, T., Brilakis, I. (eds) 2021 Proceedings of the 38th ISARC. https://doi.org/10.22260/ISARC2021/0088

Patents

Feng, C., Ergan, S., Sher, B., Xu, X., Chen, G., Javed, T., Madhusudhan, S., Mahesh, S., Vasudevan, A., Lu, D., Kiper, B. (2024). System and Method for Robotic Inspection”, https://patents.google.com/patent/US20240134007A1/

NoMoCap

Mon, 03 Feb 2020 18:47:00 +0000

Motion capture of martial arts movements have been a major component in video games and film visual effects, typically involving cumbersome suits for the martial artists and an array of expensvie cameras.

What if all you needed was a single camera? Using a computer vision model trained to recognize a person’s body parts in 2D images and locate those body parts’ 3D locations, it can be possible to generate a 3D animation of a martial artist movements using only a single video capturing those moves.

Ideally, this means there’s no more need for motion capture suits.

After takingb both a robot perception course and a computer graphics course at NYU Tandon the same semester, each of which assigned a final project, I decided to combine the two: generate 3D human figures with computer vision libraries, then animate them in 3D using OpenGL.

Full 3D pose and animation pipeline code can be found on Github, based on rayat137’s Pose_3D code for 3D human pose generation.