Julian Ceipek

Most of the world’s manufacturers rely on inadequate tools to identify and fix problems with their products: they commonly cut samples of their products in half hoping to find defects that might also exist in the units they don’t destroy. At Lumafield, I collaboratively developed approachable X-ray tools that empower manufacturers of any size to affordably find problems inside their products without destroying them. I improved every part of Lumafield’s software stack over the course of three years:

• Custom motion control firmware
• Embedded software to control and monitor a growing fleet of X-ray machines
• Touchscreen interfaces for making X-ray scans
• Server backends for ingesting, analyzing, and sharing 3D X-ray data
• Web frontends for looking inside 3D scans

While I spent most of my time on the embedded software team, I also worked closely with QA, scanner fleet services, and frontend/backend software teams to implement, troubleshoot, and fix features for customers and internal teams.

3D X-ray Vision

I joined Lumafield as the company was transitioning from early hardware and software prototypes to production-ready solutions. I developed the volumetric WebGL2 renderer at the heart of Voyager, Lumafield’s browser-based inspection tool, and prototyped improvements based on cutting-edge research papers. The renderer allows customers to look inside 3D scans, which helps manufacturers save hundreds of thousands of dollars. For example, engineers at L’Oréal used it to understand why one of their bottles was leaking, saving them 4 months of development time (see Lumafield’s case study).

This video¹ shows the renderer in action, virtually stripping away the plastic exterior of an earbud to reveal the electronics inside:

Enabling New Market Segments

I helped Lumafield expand into new market segments with high volume inspection requirements. Lumafield’s X-ray machines initially took multiple hours to scan a product. This made them unusable for high volume customers, which needed scans to complete in minutes or even seconds. I was specially selected for a small interdisciplinary (R&D, electrical, mechanical, and software) task force that produced a working tradeshow demo of fast automated scanning and analysis in one month. As part of the task force, I…

• worked closely with R&D to create milestones, time budgets, and backup plans
• analyzed and visualized each phase of the scanning and analysis pipeline to find and eliminate bottlenecks
• modified the scanner’s motion driver firmware to enable continuous scanning
• developed embedded scanner software to continuously load, scan, and unload parts while simultaneously processing scan data and streaming it to an analysis pipeline
• identified and filled knowledge and resource gaps

The following video² shows the demo in action:

After the success of the tradeshow demo, I helped turn fast scanning into a production-ready feature in close collaboration with the other two members of Lumafield’s embedded software team. Doing so required a combination of systemic redesigns, low-level optimizations, and extensive repetition testing. We used profiling tools like NSight Systems, Viztracer, and Palanteer to uncover problems; made iterative improvements; and evaluated our progress with overnight tests exposing failure probabilities when varying parameters like scan duration, resolution, and exposure time. As part of this initiative, I…

• improved scanner UI responsiveness 90% while making UI code easier to change and increasing type safety, by writing a Typescript code generator and implementing O(log(n)) instead of O(n) state updates
• reduced per-image processing time 30-50x, replacing Scipy/Numpy/OpenCV algorithms with simpler, custom SIMD C code
• rewrote the scanner’s X-ray detector hardware driver to reduce memory allocations and thread contention
• restructured scan data uploading to minimize disk usage and prioritize the most important scans
• root-caused intermittent 300ms system stalls using full-system profiling and reverse engineering tools including NSight Systems, LD_PRELOAD, Cutter, and jemalloc

This work enables Triton, Lumafield’s production-ready industrial CT machine for factories, to scan parts in under a minute.

Reducing Internal Costs

Uncompressed per-scan data occupies 10s to 100s of gigabytes and needs extensive processing to extract actionable 3D data and insights. Transferring and processing all this data can get expensive when using third party datacenters like those provided by Amazon, particularly for high-volume customers. I led a 75% variable cost reduction project that allowed us to optionally move Lumafield’s 3D reconstruction pipeline from AWS servers to on-scanner CPU+GPU compute in exchange for a relatively small increase in fixed scanner cost. During the project, I…

• established project milestones and test plans
• designed the software architecture with input from the embedded and backend software teams
• identified and delegated required hardware, electrical, and software tasks across teams
• implemented embedded software to ingest, process, and upload scan data
• deployed, monitored, and troubleshooted the rollout to Lumafield’s first high-volume customer