Shop Alto Essentials to add pharmacy products to any delivery!

Your partner in navigating the ins and outs of prescriptions — at no extra cost.

Alto Pharmacy Technician

Alto Pharmacist

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits.

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy. 

Be Part of a Great Place to Work — Join Our Team

Hear first-hand from members of the Alto technology teams.

Read the latest Alto blog posts covering healthcare, family health, managing diseases, and other Alto highlights.

Blog | Alto Pharmacy

The Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers. Read on to learn more.

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Editor's Note: The post was updated on November 8, 2021 to reflect the rebrand of AltoMD to Alto Connect.

Alto Pharmacy’s mission is to fulfill medicine’s true purpose — to improve quality of life — for everyone who needs it. The healthcare professionals who prescribe medications to Alto and the clinical teams that support them are critical partners in making our mission successful. To this end, the Providers Team at Alto is charged with building the tools and service components necessary to deliver an exceptional pharmacy experience for healthcare providers.

Every day, clinical teams -  including prescribers and other clinical staff like nurses, practice coordinators, and billing specialists - spend hours tracking down insurance information, coordinating approvals and appeals, discussing options with their patients, and playing phone tag with pharmacies. The Providers Team enables these healthcare providers to spend more time on patient care and less time on back office tasks. 

Alto’s provider communications platform, Alto Connect, and related services, including insurance benefits coordination and prior authorization management, remove clinical blockers that traditionally slow down or prevent an individual’s access to treatment. Practices that share Alto with a majority of their patients can save three or more hours per day on pharmacy and insurance administration tasks, with less time spent waiting on hold, fewer faxes and improved employee satisfaction (not to mention happier patients!). 

Anu Kashyap, a Product Manager on the team says, “I joined Alto to make a contribution to people’s health using technology, and the Providers Team is working on specific, painful healthcare problems that are critical to solve. These challenges have been incredibly fulfilling to work on.”

How does the Providers Team play into the overall growth of Alto Pharmacy?

The Providers Team is fairly new, as they were originally part of a larger Growth team within Alto. Kjersten Elias, Engineering Manager, explains: “Alto's growth model is driven by conversations about pharmacy choice in the doctor’s office, and in the fall of 2020, we dedicated an engineering team to optimizing our product offerings and improving the experience for the healthcare providers having those conversations.”

Anu adds, “The key for our business’ continued growth is having healthcare providers trust that we are the best pharmacy for their patients and leverage us as their pharmacy of choice. This type of growth channel is much more powerful than anything direct to the consumer. We can only do this however, if our providers have an excellent experience with us, and we’ve only scratched the surface here.” 

Being part of a newer team has distinct advantages. Software Engineer Dereck Quock notes, “We have unique opportunities to modernize how we build apps and define our infrastructure and technical needs. With the freedom to utilize cutting edge tech and new frameworks, there is so much room for innovation.”

What is the culture of the Providers Team like?

The engineers each raved about a small-but-important recurring calendar invite each Friday: Team Time. This protected time is a space to catch up, banter about anything other than work, and, if folks are feeling brave, face off with Anu, former Teen Tournament champion, in a rousing game of Jeopardy.

It's not hard to see why Kjersten and her team light up when they talk about team culture. Amy Chen, a Software Engineer on the team, shared that  “Kjersten is an incredible manager. Everyone is empowered to ask for and share feedback, fine-tune our processes, and check in with each other.”

“When I joined, I could tell right away this team really lifted each other up and had each other’s backs,” said Dereck. “It’s one of the greatest teams I’ve worked on.”

The Providers Team is hiring! Learn more about open positions here.

In celebration of Hispanic Heritage Month, which takes place from September 15 - October 15 every year, Alto is highlighting the important work of Alto Auténtico, our Latinx employee resource group (ERG).

Get to Know Alto Auténtico, our Latinx ERG

In celebration of Hispanic Heritage Month, which takes place from September 15 - October 15 every year, Alto is highlighting the important work of Alto Auténtico, our Latinx employee resource group (ERG)*. Below, Robert Barahona, Provider Success Specialist and co-leader of Alto Auténtico, shares more about the group’s origins, current initiatives, and plans for the future.

Robert: “Auténtico” means authentic in Spanish, and authenticity is central to the ERG’s mission. The group was created to ensure that Alto’s Latinx team members have a presence and a voice within our company culture, and that they are supported in their professional development with meaningful opportunities to grow at Alto.  

Robert: Our focus is both internal and external. In addition to developing a Latinx community at Alto, we also shape Alto’s product and services so that we’re better serving our Latinx users (more about that later). Much of that work involves building awareness of Latinx history and cultural contributions, as well as bringing attention to negative stereotypes that we want to dismantle.

We use holidays and events like Day of the Dead (November 1st and 2nd) and Cesar Chavez’ birthday (March 31) to share educational resources with our team members. On Cinco de Mayo, we hosted a Zoom happy hour and a taco lunch for all our pharmacies.

For last year’s Hispanic Heritage Month, we invited Dr. Robert Rodriguez, the president of diversity consulting firm DRR Advisors and a thought leader in Latinx talent management, to speak at Alto. He shared about his own journey of repressing his background to assimilate in the workplace to fully embracing his heritage and empowering other Latinx employees to be their authentic selves.

How are we celebrating Hispanic Heritage Month this year?

Robert: We’re focusing on celebrating Latinx intersectionality. It’s important to recognize the diversity within Latinx culture and the important perspectives of Latinx women, LGBTQ+ individuals, and those of mixed heritage, including our Black and Asian members who also identify as Hispanic or Latinx. Every week during Hispanic Heritage Month, we’re educating the full Alto team about a specific group within the broader Latinx community, with resources for additional learning.

What can team members look forward to from Alto Auténtico in the future? 

Robert: We’re really excited to continue to help shape Alto’s services and resources for customers, making our offerings as inclusive as possible. We’re currently working with the operations, product, and marketing teams to establish a bilingual team to support Spanish-speaking individuals using our pharmacy delivery services across all of our communities. 

We’re also launching a mentorship program in which Alto’s junior Latinx team members will be paired with senior Latinx team members for coaching sessions. This is another important step in continuing to invest in our Latinx team members’ professional development.

How can allies recognize Hispanic Heritage Month?

Robert: Great question! I think learning is key. Simply taking the time to educate yourself about the Hispanic and/or Latinx experience — particularly about our diversity as a group — goes a long way toward building trust, which is key to true allyship.

Interested in learning more about the Alto Auténtico employee resource group? Reach out to altoautentico@alto.com and visit alto.com/careers to learn more about our open roles. 

*ERGs are meant to empower community groups within an organization by directly linking decision makers and implementing policies that support the group’s specific needs. The groups often center around a shared identity or experience and focus on building community, providing support, and contributing to its members’ personal and professional well-being. By creating an open forum for employees to come together, discuss challenges and opportunities, and foster a sense of belonging, ERGs help both individuals and organizations thrive.

In celebration of Hispanic Heritage Month, which takes place from September 15 - October 15 every year, Alto is highlighting the important work of Alto Auténtico, our Latinx employee resource group (ERG)

Get an inside look at the patients team at Alto Pharmacy to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

Get to Know the Team That Builds Alto's Customer-Facing Apps

Imagine an app that allows you to buy things and have them delivered to you the same day, for free. Now take that app and put it in a highly regulated space like pharmacy. You face the challenge not only of building an e-commerce platform from scratch, but also of navigating medication pricing, federal and state regulations, and patient safety—and that's just the way the Patients Team likes it.

"It's a really exciting and fun problem to solve," says Mike Riess, the team's Engineering Manager. Software Engineer Angelique Nehmzow agrees: "There's a really strong product and design aspect, like more traditional startups, but then everything gets a little bit more interesting because it's healthcare."

Below, we chat with the team to learn what it's like to work on Alto's customer-facing apps, which projects are on the horizon, and who makes the best barbecue short ribs.

What does the Patients Team do at Alto? 

Ask anyone on the team why they joined, and chances are, they'll tell you they wanted to be on the product team closest to Alto's customers: "We really try to own the customer experience," says Peter Zeng, the latest Software Engineer to join the team. "If a customer is coming to the web app, or they're coming to our mobile app, we want to make sure that they're able to onboard with us as easily and conveniently as possible."

The team is also responsible for the customer experience beyond onboarding, from the moment a customer schedules a medication for delivery to the time that it shows up on their doorstep. "We already offer great services like free, same-day delivery and cost savings. The Patients Team empowers customers to make the most of our offering," Nehmzow explains. "You can see what prescriptions you have, check on the status of your order, and schedule automatic refills. In a more traditional pharmacy, you don't get that kind of visibility. You have to call them up or even wait in line in person. At Alto, it's a different experience."

What was it like in the team's early days?

Today, there are eleven engineers on the team working to build that better end-to-end experience—but it wasn't always that way. When the team first formed in 2019, it consisted of just three members: Riess, Brian McRae, and Andy Lau, all of whom are still on the team.

"The first priority for us was just stabilizing the foundations," Riess says of the team's early days. Every other week, the three team members, along with Alto co-founders Matt Gamache-Asselin and Jamie Karraker, would meet to audit the app's core flows. "For the first three months we probably didn't make it through some flow at least 50% of the time," Riess recalls. "There would be a ton of bugs in the onboarding experience, things that weren't working with the flows or navigation or deep linking."

Among the team's first projects was a redesign of the checkout flow. "There was a lot of work to be done to streamline the checkout flow, which is pretty much the main function of the app," says McRae, who initially joined the team as a software engineer and later transitioned into product management. "We had to make it easier to add a payment, add your addresses, everything you need to do to schedule your prescription for delivery."

"The growth of the team, in terms of both size and stability of the platform, has allowed us to swing at more interesting bets for the company," Riess says. In particular, the Patients Team has turned their attention toward experiences personalized for specific medical conditions.

"Different customers have different needs. A customer who wishes to start a family will have a different experience than a customer who needs to order their routine heart medications that they've been taking for the last eight years,” McRae says. “The challenge for us is how we serve all these customers well through the app."

"I think that's where Alto can be such a differentiator," Riess adds. "A world-class pharmacy experience is about so much more than just dispensing medications. We want to build tools for customers that help them get the most out of their medications—everything from simple videos teaching them how to perform injections safely at home to the ability to synchronize deliveries for multiple prescriptions for managing complex health conditions."

With these new projects on the horizon, engineers have ample opportunity to focus on the areas that excite them most. "We're full-stack and mobile and web," Nehmzow says. "Part of the team focuses more on experimentation, iterating quickly, and using data to make decisions. Part of the team focuses more on platform work, tooling, and improving the developer experience. There are a lot of areas to explore and contribute to."

One spring night, the Patients Team gathered in a back yard in San Francisco’s Mission District for short ribs and an impressive spread of side dishes. "Kudos to Robbi, who's the grill master. I just provide the grill!" Nehmzow says, laughing. While the team barbecue was the first time some members had met in person, Zeng notes that the team has been supportive of one another even while working remotely during the pandemic.

"The first real project that I was assigned to was working with the communications system. Before writing my first piece of code, I spent two weeks exploring what already exists and understanding the different parts of the system," Zeng says. "I've been really impressed with how understanding my teammates have been. They make the space for you to learn all the different pieces of code we own."

That culture of support is by design, reflected in the six principles—complete with matching emojis—at the top of the #patients-team Slack channel: "Empower each other," "be an owner," "find the path," "learn by doing," "embrace complexity," and "foster trust." 

While these principles echo Alto's company-wide values, they are unique to the Patients Team. "For a long time, we were the only external-facing product team at Alto," Riess says. "These principles were intended to be this motivation around how we win. It's a combination of how we treat each other, how we reason about decision making, and how we approach solving a problem that's never existed before."

As the team looks toward the future, those principles have never been more relevant: "We have a goal and an area we want to focus on, but in terms of which projects we're going to do, it's not a top-down thing," Nehmzow says. "It's very much, 'We're going to figure out how to solve this problem together, even when it's difficult.' That's my favorite part about the team."

The Patients Team is hiring! Learn more about open positions here.

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies.

Alto Pharmacy Named Great Place to Work®

It’s official: Alto Pharmacy is a great place to work! We’re honored to join the community of Great Place to Work-Certified™ companies based on team member feedback and a rigorous assessment of our company’s culture and benefits. This important recognition is a testament to the dedication of the entire Alto team and the vision of our founders to create a healthcare company truly anchored on the needs and experiences of the customer.

80% of our team says that Alto is a great place of work, compared to just 59% of employees at  other companies that participated in this year’s assessment. Here are other key takeaways from our team’s’ responses:

93% of our team members say that you are made to feel welcome when you join Alto.

88% say that people care about each other here.

86% say that they are able to take time off from work when needed.

Every company has its own unique culture, but there are two things that all great workplaces share: trust in company leadership and a strong sense of pride in the work. For many Alto team members, being part of a mission-driven company is one of the best parts of working here. 

Our mission — to improve the quality of life for everyone who needs medicine — guides us in every business decision, every day. From fostering an inclusive culture that invests in talent from the same communities as our patients, to building a pharmacy model that meets individuals where they are in their health journey, everything we do at Alto is anchored on delivering meaningful change. 

When asked what makes Alto a great place to work, our team members emphasized our guiding company values — including our commitment to putting patients first and leading with empathy — and getting to spend each workday with colleagues who are also inspired and motivated by those values.

“Our bold, patient-centered mission creates a powerful level of empathy across the entire team,” shared Rob Anderson, Alto’s Head of Talent . “The people who work here are one of the best parts of coming into work, and one of the top reasons our workplace is so rewarding.”

And taking care of our team members is just as important as ensuring we leave no patient behind. In response to the past year’s unprecedented challenges, we introduced Alto Recharge Week, which includes expanded time off for all team members. This reset period allows team members to unplug and recharge, and it’s also a small show of  appreciation for how they supported our patients and healthcare providers throughout these challenging times.  

It’s an exciting moment to be part of Alto’s mission. Since our founding in 2015, we’ve been focused on delivering a totally new model of pharmacy care. Our community-based, courier-first pharmacy model simplifies the entire prescription process and is tailored to each patient’s unique needs, leading to better  outcomes, healthier lives, and happier customers. And with every year, Alto’s impact continues to grow:

More than 36 million individuals now live within Alto’s free same-day courier service zones. 

In the past twelve months, we opened new locations in New York City, Houston, Dallas, and San Jose and expanded our coverage areas in Seattle, Las Vegas, and Denver. 

We've expanded our specialized resources for patients living with diabetes and managing heart and lung conditions .

We’ve accomplished so much over the past six years, and we’re still just getting started.

“I don’t know a single person who hasn’t needed something from a pharmacy at some point, and there’s still so many people across the country who have yet to experience our unique community pharmacy experience,” Alto’s Recruiting Manager Lauren Binet said. “That potential for impact is incredibly motivating.”

As our geographic footprint has grown, so has our team. In the past two years alone, we’ve quadruped our team to more than 1,000 Altoids!

Join us in delivering our industry-leading pharmacy care to millions of people across the country. As a member of the team, you’ll play an integral role in improving our customers’ quality of life and redefining what’s possible in pharmacy.

View open roles in design, engineering, IT, sales, patient care, and more.

About Great Place to Work Certification™

Great Place to Work® Certification™ is the most definitive “employer-of-choice” recognition that companies aspire to achieve. It is the only recognition based entirely on what employees report about their workplace experience – specifically, how consistently they experience a high-trust workplace. Great Place to Work Certification is recognized worldwide by employees and employers alike and is the global benchmark for identifying and recognizing outstanding employee experience. Every year, more than 10,000 companies across 60 countries apply to get Great Place to Work-Certified.

All third party trademarks (including logos and icons) referenced by Alto Pharmacy remain the property of their respective owners. Unless specifically identified as such, Alto’s use of third party trademarks does not indicate any relationship, sponsorship or endorsement between Alto and the owners of these trademarks.

In recognition of Women’s History Month, we highlight four women at Alto shaping the future of pharmacy. These leaders share their advice for women in the workplace, their unique super powers, and what they appreciate most about Alto's culture.

Celebrating Women at Alto Pharmacy

Liz Kelly, Sr. Director Sales, Fertility 

I'm responsible for hiring, coaching and supporting the needs of the national fertility sales team. I also work closely with fertility business and operations teams to deliver on our clinic and customer promises.

Before joining Alto, I was finishing up a sabbatical where I spent my time volunteering on the board of a non-profit called ServeHAITI. Prior to that, I was the West Coast Area Business Director at EMD Serono and worked with Matt and Jaime [Alto’s co-founders] to secure a contract to distribute fertility meds when they were still working out of AG Pharmacy!

What do you appreciate the most about Alto's culture? 

Hands down my favorite part of being on the Alto team is that the patient is truly at the center of every decision we make. I value that we make business decisions with our mission in mind.

What do you like doing outside of work?  

Of late, my favorite pastime is heading to the dog beach with my rescue, Bodhi, who has been the best medicine to ease some of the COVID-19 isolation I experienced over the past year. Before the pandemic, I made sure to travel to at least one new international spot a year. I love the experience of getting exposed to all things about another culture: cuisine, culture, belief systems, language.

I like to think that my super power is seeing the greatness in others, sometimes before they see it in themselves.

I aspire to live by the motto "be the change you wish to see in the world." I encourage women to speak up for what they believe is right, even if nobody else agrees. That includes planning for your own career path and making it a priority to develop your skill sets so that you are prepared to climb the ladder that serves your own greatness.

I graduated from pharmacy school in May of 1979 and started working at AG Pharmacy after passing my boards later that year. I bought AG in 1998 when the previous owner wanted to retire - he had been with AG since it first opened in the early 60's.

My family moved to San Francisco from Hong Kong when I was five years old. Our story is one similar to those of many immigrant families who came to America looking for a better life in search of the American dream: own a home, have a car, raise children safely and give them the opportunity of a good education and a chance to have a better life. My parents raised their children here, emphasizing family, hard work and education. They lived their American dream, and I have them to thank for all that I am today.

What’s your favorite thing to do outside of work? 

I love to travel. There are so many unique places to visit in this world. With the different cultures, the food, and the amazing sceneries/landscapes, there are endless things to see and experience! 

The thing I appreciate most about Alto's culture is that everyone works so hard together to bring about the best pharmacy experience for all our customers. You can feel the energy and the motivation of all the Altoids. I cannot emphasize how amazing it is to be a part of this team and to watch Alto grow and evolve into the pharmacy of the future and the pharmacy of our dreams!

What’s your favorite memory thus far at Alto? 

In the very early days, when Alto was still Scriptdash Pharmacy, a customer needed to get her trigger shot [as part of a fertility protocol] within the next few hours. It was close to rush hour, and we had no couriers available to deliver to the South Bay. Our CEO Matt came to the rescue on his motorcycle, and off he went to make the delivery. It was such a wonderful and warm feeling to watch and cheer him on together. That is truly what Alto love is all about!

I manage our amazing IT Support team, helping all Altoids get the technology support they need, when they need it. 

For the better part of the last decade I have been managing IT Support teams. From Facebook and Twitter to Lyft and Uber, I’ve seen it all (or at least most of it!).

Our incredible dedication to our mission. In my short time here, I truly see us putting patients first, and it’s moving to see. 

What do you like doing outside of work? 

I’m a big traveler, but since COVID-19 has put a damper on that, I enjoy flying around Colorado with my partner in our vintage Beechcraft Bonanza. I also love hopping on my motorcycle and just exploring! 

I’m in love with Central and Eastern Europe. Some of my favorite places are Prague, Vienna and Budapest. 

I’m originally from Iran but moved a lot growing up, including Oklahoma, Australia, Qatar, Kuwait, and Chicago. I have been in San Francisco for the last six years. I stumbled across software engineering in college and found a passion for building products and teams. I joined Alto as the third engineer and this is now the largest company I’ve ever worked at. 

I lead a group of 20 engineers across three teams. Two teams are focused on getting prescriptions ready to schedule quickly and accurately, and one is focused on enabling faster, higher quality, and more enjoyable product development across the broader product and technology organization. My work involves a lot of collaboration, creativity, transparency, and flexibility. 

I was an early software engineer at a small startup; before that I was a hybrid software engineer and product manager at a design and development studio. 

I like how passionate everyone is about what they do. 

I’m at my best managing a crisis or finding a path through conflicting strong opinions. 

I want to start with some advice for our allies: get more curious about the experience of the women you work with and learn about ways you can support and amplify the work of the women around you. Diversity, equality and inclusion is a team sport and, even if we win a game or two, it’s about the continuous pursuit of excellence. 

For my fellow women at Alto, I wish I knew earlier to focus more on improving my craft and impact so I can get a seat at the table, rather than trying to juggle advocacy and my own work. Once you’re at the table, you can really advocate for hiring, growing, and retaining women. 

To learn more about the work of our six employee resource groups (ERG’s) focused on building community - including Umoja (Black), Auténtico (LatinX/Hispanic), Women (WOAA), Proud (LGBTQIA+), Asian/Pacific Islander, and Working Parents - read our 2020 Diversity, Equality and Inclusion Report. 

Are you passionate about shaping the future of pharmacy? Join us as we deliver our industry-leading pharmacy care to millions of Americans. 

Alto Pharmacy is recognized by Forbes as one of America’s Best Startup Employers in 2021, ranking 13th among 60 healthcare startups.

Alto Pharmacy Announced as One of Forbes Best Startup Employers

After discovering that half of all prescriptions aren’t even picked up—resulting in 125,000 avoidable deaths every year—we set out to build a new kind of pharmacy that focuses on the person behind every prescription.

Here at Alto, we’re transforming a $500 billion industry by reinventing what a pharmacy can do—from hand-delivering medications for free to offering on-call help by text or chat. Our customers are at the center of everything we do, and it shows. After delivering more than one million prescriptions, we have an NPS score of 86 and over a thousand five-star reviews on Yelp.

In addition to delivering a better pharmacy experience for everyone who needs medicine, we’re committed to cultivating a better workplace for for our team, every day. That’s why we’re honored to be recognized by Forbes as one of America’s Best Startup Employers in 2021, ranking 13th among 60 healthcare startups. Inclusion on the list was based on professional reputation, employee satisfaction, and growth, among other criteria. 

This accolade follows several others in recent months including LinkedIn’s Top Startups in 2020 and San Francisco Business Times Largest Bay Area Digital Health Companies in 2020. In addition, Alto’s CEO, Matt Gamache-Asselin, recently received recognition as one of The Top 25 Consumer Health Tech Executives of 2020 by The Healthcare Technology Report. 

“We have a lot to be proud of in the past year, especially in light of the challenges presented by the COVID-19 pandemic,” shared Gamache-Asselin. “I’m so proud of the hard work and commitment exhibited by every member of our team in providing exceptional pharmacy care in our communities. As we continue to expand our services to more and more people, we’ll also be growing our team of incredibly knowledgeable and caring professionals.”

Are you passionate about shaping the future of pharmacy? Join us as we  deliver our industry-leading pharmacy care to millions of Americans. 

For the full employers list and an overview of the selection methodology, head to the Forbes site. .

Alto Pharmacy Announced as One of Forbes Best Startup Employers 

Forbes 2021 America's best startup employers

Justin Moy, Senior Software Engineer at Alto, explains how the engineering team developed a staging environment strategy that could keep up with our growing organization.

Creating Sandboxes: How to Make It Easy for Your Engineering and Product Teams

When we formed our Infrastructure team in 2020, our charter centered on scale, reliability, and the developer experience. Our first priority? Developing a staging environment strategy that could keep up with our growing organization.

Initially, we had a set of command-line tools for staging environments that could be run from an engineer’s workstation. However, this approach required significant manual effort to create and manage an environment which led to inconsistencies like outdated data and lack of full application coverage and integration. The required workstation dependencies and maintenance added to the overhead. These tools were also different from the ones used to create a production environment, causing drift between the two.

To address these issues, we focused on improving our tooling to create consistent and reproducible results for both our staging and production environments. Here are the five steps we took to get there:

Luckily, we were already here. Our deployment process involved creating and deploying Docker containers and using a Procfile to run specific parts of our applications (think separating out web and workers). Packaging our applications as immutable Docker images allowed us to ship consistent and portable code. These images are run as containers which can be executed on separate hosts and scaled independently of each other.

We wanted to increase our observability, mature our internal monitoring, and maintain our infrastructure in a more consistent and reproducible manner. Kubernetes, and its automated container orchestration (deployment, scaling, and management), allowed us to accomplish all of the above goals.

It was easy for us to get started in Kubernetes thanks to Kind, which allowed us to run clusters locally. This increased development speed and reduced the friction of learning a new external provider.

Helm makes it easy to manage packaged applications in Kubernetes. Some community-provided applications include: NGINX for load balancing and reverse proxying HTTP and other traffic; Prometheus and Cortex for metrics, monitoring, alerting, and dashboarding, Loki for logs; and Sentry for exceptions.

In order to ease the integration with other infrastructure components, enable rapid deployment, and, most importantly, provide us with the consistency and reproducibility we wanted, we created and deployed Helm charts for our applications.

From here, we could stamp out our applications for developer use in sandbox environments: complete end-to-end environments, separate from production, that enable engineers to rapidly develop and test their code. And if an engineer is working on multiple features, they can spin up multiple sandbox environments and manage their applications and data independently of each other. Victory!

We had already created pipelines in Concourse to automatically deploy and upgrade our infrastructure Helm charts. To eliminate the manual toil of spinning up new environments and to reduce the friction around sandbox adoption for our engineers, we extended these automated pipelines by building a Git-driven continuous deployment pipeline. When an engineer created a commit to a repo defining the state of a sandbox, our Concourse job would apply these changes.

At this point, we could deploy environments that resembled production but were separate from production and other sandbox environments. They connected to the rest of our observability stack, including metrics, logs, traces, and exceptions.

As we prepared to scale to the rest of the product and engineering organization, we focused on the following ideas: further reducing manual toil, removing barriers to entry, reducing cost, and optimizing the development workflow. The diagram below shows our sandbox architecture.

In Kubernetes, an operator acts on custom resources in order to automate a task or set of tasks. The custom resource allows us to create a well-defined entity and extend configurability options to users. The operator allows us to customize individual sandbox environments based on changes to the custom resource.

We utilized Kubebuilder to create our operator and CRD (custom resource definition). Kubebuilder was a great starting point and provides tools to ease rapid development of a new operator.

Using this pattern, we could easily perform CRUD operations (create, read, update, delete) on a custom resource using standard tools like kubectl, and the operator would handle updating our deployments in Kubernetes. These updates could include: updates to application configuration, deploying a new Docker container, injecting different secrets, etc. This flexibility allows for testing of many different scenarios like: developing a new feature, reproducing a bug, upgrading frameworks, and security and load testing.

Increase adoption by removing barriers to entry (admin web UI)

We built a web UI so users would be able to deploy sandboxes intuitively and easily without extensive knowledge of the underlying tools. The web UI interacts with our custom resource and lets the operator handle updates. It allows our teammates—both engineers and non-engineers—to easily spin up their own environments for development, demo, or training purposes without modifying production data.

In addition, we show helpful information about a sandbox to a user. This includes links to each of our applications; pre-filtered logs, metrics, tracing, and exceptions; and commands to connect to specific applications to ease debugging.

Because the sandbox environments are integrated with our monitoring stack, we have the full metrics and logging that we expect from our applications. We created an autoscaler service that performs queries against this information and scales our sandbox environments up and down. We’re able to decrease our costs by over 90% when these environments are not in use and are able to quickly scale them back up when they are being used.

Optimize the development workflow (image updater)

An additional feature we built into our sandbox environment is the ability to follow a feature branch in our applications. When an engineer pushes a commit to a pull request in Github, a corresponding sandbox will automatically update to this commit. This reduces several manual steps in the development workflow and allows engineers to test their code in near-to real time.

Sandbox environments have made an impact on the engineering organization and beyond.

Engineers can quickly desk check new code in a production-like environment during code reviews.

Shared sandboxes reduce the blast radius of bug bashes.

Non-engineers can spin up sandboxes to demo or provide training on features without impacting production data via the Admin UI. This has proven particularly useful for Product Managers and Designers on the team to be able to play with product features and better understand how they work in order to suggest improvements.

Teams have built use case scenarios into their applications. This allows easy reproducibility of critical paths and functionality.

With these five steps—Containerize, Kubernetes-ize, Helm-ify, Automate, Optimize—we’ve laid the foundation for deploying all of our environments in an automated and consistent manner. In addition to applying many of our learnings to improve our production environment, we successfully built core infrastructure that our engineers, product managers, and designers use each month to test their changes and reduce production downtime.

Interested in learning more about engineering at Alto? Follow us on LinkedIn. 

In this post, we'll share how we identified the problem of failing cypress seeds, what the culprit was, and three ways to make seeding test data more robust.

How to Debug Slow and Failing Cypress Seeds

End-to-end test failing before it even begins? Here’s how we timed, debugged, and sped up our seed data.

At Alto, we use Cypress to test our Rails application end-to-end. As our application grows, so does our end-to-end test suite — along with the seed data it requires. This led us recently to a perplexing problem:  when run in CircleCI, our end-to-end tests would mysteriously time out during the seed step, causing them to fail and block deploys.

Seeds can be painful to debug. They’re typically set up through a single request and, depending on how thorough the tests are, call upon most of the models involved in the application. If a model throws, say, a validation error because of bad seed data, you'll see it in your Cypress logs. If a model hangs, the request often doesn't show up in the logs at all. There's no status code, no error message—all you know is that somewhere, somehow, some process is failing to finish.

In this post, I’ll share how we identified the problem, what the culprit was, and three ways to make seeding test data more robust.

To Identify the Problem, Focus on Your Logs

We started by comparing logs from our failing CircleCI runs with those from local test runs. Turns out, our CircleCI logs in failed runs stopped after a specific log message. 

For example, when our test runs succeeded, the logs looked something like this:

...
LOG: creating model
WARN: foo with id=123 not found
WARN: foo with id=456 not found
WARN: bar with id=789 not found
ERROR: Invalid parameter to function `foobar`
LOG: model created
... rest of logs

But when they failed, they instead looked like this:

. . .
LOG: creating model
WARN: foo with id=123 not found
WARN: foo with id=456 not found
[end of logs]

This let us know where to focus: after the function that looks for foo, but before or during the one that looks for bar. Of course, this only works if you have enough log statements scattered around in the first place. I’d recommend logging the time it takes to create each model in seeds, which has the added benefit of telling you where to start if your seeds are becoming annoyingly slow. 

At the top of our file in db/seeds, we declare the following function to add timings to our CI logs:

@last_time = Time.now
def log_time(label)
    Rails.logger.info("Time [#{label}]: #{(Time.now - @last_time)}")
    @last_time = Time.now
end

We then call the function after each create statement in our seeds:

Product.create!({...params})
log_time(“Product Create”)

This prints logs in the following format:

...log metadata message=Time [Product Create]: 12345
...log metadata message=Time [User Create]: 1102
...log metadata message=Time [Address Create]: 105
...

These logs will help you figure out problematic model create calls, but each model may have a lot going on in its callbacks (before_save, before_commit, after_save, etc). You can follow a similar approach to time those callbacks:

# in the model file
Class Model < ApplicationRecord
	
...

before_save :foo

...

def foo()
   start_time = Time.now
   [ ... things that happen inside the function ]
   Rails.logger.info("Time [foo]: #{(Time.now - start_time)}")
   [ function return statement, if applicable ]
end

It’s important to note that the most time-consuming statements in your seeds often cause the most trouble, but that’s not always the case. If your seeds are hanging due to external API calls, it’s entirely possible for those calls to usually take a short time to complete. If you have hanging tests, I would recommend logging test runs in CircleCI, since you may not be able to find the problem locally. 

Through timing our seeds, we narrowed down our search to five of our most time-intensive models. We were then able to time the individual callbacks within those models. One callback  in particular—containing a call to an external database—raised the most suspicion. It took up most of the time it took to create the model, didn’t return any information needed in our test, and was not mocked or stubbed out. 

Requests to this test database would usually work, however, since we call on it multiple times on every CI run, with concurrent CI runs, these requests would sometimes take a long time to complete, leading to a test timeout. 

In our case, since we were making calls to a hosted database in our tests, we changed our test environment to create and refer to an isolated instance of the database. While this was the best approach for our specific issue, we also explored some other ways of stubbing out API calls and skipping callbacks from your seeds file, which we describe below. Use these approaches with caution, as the more you stub, the less true-to-life your tests will be. 

Ideally, you would have already stubbed out third-party API calls in your code using environment variables. An example approach:	

def do_the_thing()
    if Rails.env.production?
    	[ ...do the thing ]
    else
	[ ...do some fake thing if needed ]
       Rails.logger.info("We theoretically would have done the thing.")
end

However, you can also use Typhoeus to mock third-party api calls in seeds, similar to how you might in a test. Imagine that your Product model makes a call to "https://slack.com/api/foo”. We can then mock that call out as follows:

mock_response = Typhoeus::Response.new({code:’500’, body: {
    “ok”: false,
    “error”: ‘not_authed’
}.to_json})
Typhoeus.stub("https://slack.com/api/foo”).and_return(mock_response)
Product.create!({...params})

Again, I would advise you to stub check for environment variables in your code rather than stub out individual API calls in your seeds and tests. But if for some reason you need to do the latter, it’s possible!

Similarly, you might want to stub out a function you’ve written if it takes a long time to run and doesn’t affect the outcome of your tests. This can’t be stubbed out in the ways we mentioned above since (1) you might need it  for — and hopefully are testing it in  — other tests and (2) it doesn’t have a url. minitest has a tool for this. Imagine you’re trying to stub out some functions from a Photograph model.

require ‘minitest/autorun’

Photograph.stub :fetch_product_photo_url, ‘’ do
   Product.create!({...params})
end

This will return an empty string when Photograph.fetch_product_photo_url is called within the Product model’s callbacks. You can also replace the empty string with anything else you might want the function to return.

Last but not least, it’s possible to skip entire callbacks and validation. I wouldn’t recommend skipping all of them, as that can cause models in your tests to stray very far from how your models actually work, but you may find it useful to skip specific callbacks.

While all the methods that I’ve suggested above are possible and can be useful, remember that stubbing and skipping from your seeds may just be band-aids for issues you have elsewhere. In our case, we could have stubbed out a model function or created an isolated database for testing. While both methods would have been functionally the same, doing the latter allows us to have to worry less about that database in the future—if, for example, we write more functions that depend on it, or update the mocked functions to do more than they currently do. Keeping tests in line with how the product works also keeps it more in line with the actual user experience: if it’s taking a long time to create a model in your seeds, it probably takes a long time for your users, too. 

Since adding timing logs and creating an isolated database to our end-to-end tests, we have a regular source of truth for the efficiency of our models, are able to develop more smoothly with fewer spontaneous test failures, and have more confidence in our ability to quickly pinpoint and resolve seed timeouts in the future. 

We were honored to sit down with Dr. Hildreth to discuss the challenges and opportunities we are facing at this pivotal moment in history. In addition to recounting his inspiring personal story, he shared many profound insights about the COVID-19 pandemic and the state of American healthcare. 

The COVID-19 Vaccine & Racial Disparities in Healthcare: A Q&A with Dr. James Hildreth

In August, we announced that Dr. James Hildreth had joined our newly formed Advisory Board as one of its first two anchor members. Dr. Hildreth’s extensive healthcare expertise has been invaluable as we’ve sought to guide Alto into the future and continue reimagining the pharmacy experience. 

Blog

Get to Know the Team That Has Doctors Calling Alto “The Pharmacy of The Future”

Get to Know Alto Auténtico, our Latinx ERG

Get to Know the Team That Builds Alto's Customer-Facing Apps

Alto Pharmacy Named Great Place to Work®

Celebrating Women at Alto Pharmacy

Alto Pharmacy Announced as One of Forbes Best Startup Employers

Creating Sandboxes: How to Make It Easy for Your Engineering and Product Teams

How to Debug Slow and Failing Cypress Seeds

The COVID-19 Vaccine & Racial Disparities in Healthcare: A Q&A with Dr. James Hildreth

Supporting Working Parents at Alto

Two Engineers Crunch the Numbers: When Is It Cheaper to Pay Cash for Your Medication?

Get to Know Alto’s VP of Engineering, Mike McClay

Switch to a better pharmacy.