Goevry World Shopping Magazine

Top Menu

  • DMCA
  • Privacy Policy
  • Contacts
  • UK
  • DE

Main Menu

  • Travel & Lifestyle
  • Fashion
  • Health & Beauty
  • Science & Tech
  • Gift Guides
  • Buying Guides
  • DMCA
  • Privacy Policy
  • Contacts
  • UK
  • DE

logo

Goevry World Shopping Magazine

  • Travel & Lifestyle
    • Vom Bildschirm zum Bildschirm: Memmo Bringt Promis Direkt Zu Dir

      July 18, 2025
      0
    • Entdecken Sie die Freude am nahtlosen Reisen: Die Kraft von Deutschlandticket entfalten

      July 10, 2025
      0
    • Wählen Sie die beste Fähre für Ihre nächste Reise: Starten Sie in ...

      July 10, 2025
      0
    • The QHotels Collection – Elegant Escapes, Golf Retreats, and Relaxed Luxury Stays ...

      July 9, 2025
      0
    • Erkunden Sie Europas atemberaubendste Städte bequem und entspannt bei Ihrem nächsten Städtetrip

      July 9, 2025
      0
    • Caesars Rewards – The Ultimate Destination for Gaming, Luxury, and World-Class Hospitality

      July 9, 2025
      0
    • Discover Xcaret: Where Nature, Culture, and Adventure Come to Life

      July 9, 2025
      0
    • One World Observatory: Rise Above, See Beyond, Feel New York Like Never ...

      July 9, 2025
      0
    • Unveiling the Post Office: Unlocking a World of Convenience and Services

      July 9, 2025
      0
  • Fashion
    • Erleben Sie den Luxus von Sisley Paris: Premium-Hautpflege und Make-up für ein ...

      July 11, 2025
      0
    • Empowerment Starts Underneath: The EBY Revolution in Comfort and Purpose

      July 9, 2025
      0
    • Shop Ethical Fashion and Vintage Finds to Support Global Causes While Elevating ...

      July 9, 2025
      0
    • Wear the Change – Explore Comfort and Purpose with EBY Intimates

      June 26, 2025
      0
    • Seamless Confidence – Redefining Everyday Comfort with EBY Intimates

      June 20, 2025
      0
    • Stylish Finds for Little Ones: Shop Pre-Loved Kids' Dresses with Purpose

      June 13, 2025
      0
    • Jeden Kilometer meistern: Laufbar Laufjacke für Herren – Unübertroffener Komfort, Stil und ...

      June 10, 2025
      0
    • Scopri profumi femminili di lusso per ogni stato d'animo e occasione

      June 6, 2025
      0
    • Shop Smart, Dress Sharp: Ethical Superstore Men’s Shirts for a Stylish and ...

      June 4, 2025
      0
  • Health & Beauty
    • Erschließen Sie Ihr Potenzial mit Forever Young: Eine Reise zu Gesundheit und ...

      July 12, 2025
      0
    • Choose the Right Fragrance for You: Top Perfumes That Define Femininity and ...

      July 9, 2025
      0
    • Fuel Your Workout with MyProtein’s Protein Powders: Amazing Offers Inside

      June 12, 2025
      0
    • Verwandeln Sie Ihr Haar mit MiiN Cosmetics: Die ultimative Revolution in der ...

      June 11, 2025
      0
    • Verbessern Sie Ihre Wellness-Routine mit Lucky Hemp CBD Ölen

      June 11, 2025
      0
    • L'ARISÉ Beauty: Entdecken Sie den Reiz von pudriger Eleganz und zeitloser Raffinesse ...

      June 9, 2025
      0
    • Scopri i prodotti delicati per la cura dei capelli e per ottenere ...

      June 8, 2025
      0
    • Trasforma la tua pelle con questi prodotti idratanti essenziali – Sconti speciali

      June 6, 2025
      0
    • Le migliori soluzioni per la cura della pelle: protezione solare e sollievo ...

      June 6, 2025
      0
  • Science & Tech
    • Scoprite il futuro dell'informatica con i portatili e le workstation ad alte ...

      June 15, 2025
      0
    • Entdecken Sie die Innovation hinter Ninja-Geräten: Revolutionieren Sie Ihre Küche noch heute

      June 2, 2025
      0
    • Unleash Your Imagination: Explore Science Fiction & Fantasy Audiobooks with Audible

      March 12, 2025
      0
    • Apple MacBook Air bei WIRKAUFENS - Clevere Angebote, Top-Qualität und ultimative Leistung!

      March 1, 2025
      0
    • AGM-Batterien bei ATP Autoteile - Kraft, Leistung und Zuverlässigkeit für Ihr Fahrzeug!

      February 26, 2025
      0
    • Crush Your Hiring Test with JobTestPrep – Your Gateway to Career Success

      February 25, 2025
      0
    • Master Every Job Assessment with JobTestPrep Comprehensive Practice Tests and Expert Study ...

      February 19, 2025
      0
    • Transformieren Sie Ihr Unternehmen Mit Sage: Effizienz In Höchstform

      January 31, 2025
      0
    • Alla scoperta dell'Universo con l'Istituto Stellare Comprendere i misteri del Cosmo

      January 9, 2025
      0
  • Gift Guides
    • Create Lasting Memories with CEWE Photo Book: Your Personalized Keepsake from Boots ...

      May 8, 2025
      0
    • Farrar & Tanner: Luxury and Bespoke Gifts for Every Occasion

      March 25, 2025
      0
    • Tommee Tippee Soothers: The Perfect Blend of Comfort, Style, and Practicality for ...

      March 13, 2025
      0
    • Indulge at the Pinnacle of Private Luxury with Je Joue

      March 8, 2025
      0
    • Get Closer to Your Favorite Stars With Memmo.me’s Exclusive Celebrity Video Messages

      February 25, 2025
      0
    • Waterford Crystal – Mastering the Art of Fine Glassware and Elegant Home ...

      February 15, 2025
      0
    • Discover the World of Exclusive Whiskies with The Scotch Malt Whisky Society

      February 8, 2025
      0
    • Embark on an Exclusive Whisky Journey with The Scotch Malt Whisky Society

      February 8, 2025
      0
    • Waterford Crystal – Timeless Beauty and Artistry in Luxury Glassware and Home ...

      January 28, 2025
      0
  • Buying Guides
    • Elevate Your Game with Premium Golf Balls and Exclusive Deals at Discount ...

      April 17, 2025
      0
    • Aatu’s Premium Dog and Cat Food Delivers High-Quality Nutrition with Natural Ingredients

      April 14, 2025
      0
    • AATU Offers Premium Pet Food Made with Fresh Meat and Natural Ingredients ...

      April 9, 2025
      0
    • Discover the Best Boxsets from Townsendmusic

      February 21, 2025
      0
    • Transform Your Garden into a Bird Haven with These Must-Haves

      January 10, 2025
      0
    • Discovering Unique Whisky Flavours: A Journey of Taste and Tradition

      January 10, 2025
      0
    • Stress-Free Parenting Explore Tommee Tippee’s Range of Baby Care Innovations

      December 26, 2024
      0
    • TOMMEE TIPPEE Innovative Designs for Happy, Healthy Babies and Stress-Free Parents

      December 12, 2024
      0
    • Intelligent einkaufen: Der ultimative Preisvergleich für die besten Deals

      December 11, 2024
      0
  • Vom Bildschirm zum Bildschirm: Memmo Bringt Promis Direkt Zu Dir

  • Experience Baseball More Than Just a Game: Step Up to the Plate with New York Yankees’ Legendary Fan Experience

  • Ultimate New York Yankees Memorabilia: Show Your Team Pride Today!

  • Feel the Pulse of Baseball’s Greatest Stage: Score Big with New York Yankees Tickets for an Unforgettable 2025 Season

  • Erschließen Sie Ihr Potenzial mit Forever Young: Eine Reise zu Gesundheit und Vitalität

  • Erleben Sie den Luxus von Sisley Paris: Premium-Hautpflege und Make-up für ein makelloses Aussehen

  • Drive Smart, Travel Far – Discover Flexible, Eco-Friendly Car Hire with Europcar Worldwide

  • Europcar – Reliable Car Rentals With Flexible Plans and Hidden Fee Risks

All
Home›All›Dodging the data bottleneck — data mesh at Starship | by Taavi Pungas | Starship Technologies

Dodging the data bottleneck — data mesh at Starship | by Taavi Pungas | Starship Technologies

By admin1
May 11, 2021
261
0
Share:

[ad_1]

A gigabyte of data for a bag of groceries. This is what you get when doing a robotic delivery. That’s a lot of data — especially if you repeat it more than a million times like we have.

But the rabbit hole goes deeper. The data are also incredibly diverse: robot sensor and image data, user interactions with our apps, transactional data from orders, and much more. And equally diverse are the use cases, ranging from training deep neural networks to creating polished visualizations for our merchant partners, and everything in between.

So far, we have been able to handle all of this complexity with our centralized data team. By now, continued exponential growth has led us to seek new ways of working to keep up the pace.

We have found the data mesh paradigm to be the best way forward. I’ll describe Starship’s take on the data mesh below, but first, let’s go through a brief summary of the approach and why we decided to go with it.

What is a data mesh?

The data mesh framework was first described by Zhamak Dehghani. The paradigm rests on the following core concepts: data products, data domains, data platform, and data governance.

The key intention of the data mesh framework has been to help large organizations eliminate data engineering bottlenecks and deal with complexity. Therefore it addresses many details that are relevant in an enterprise setting, ranging from data quality, architecture, and security to governance and organizational structure. As it stands, only a couple of companies have publicly announced adhering to the data mesh paradigm — all large multi-billion-dollar enterprises. Despite that, we think that it can be successfully applied in smaller companies, too.

Data mesh in Starship

Do the data work close to the people producing or consuming the information

To run hyperlocal robotic delivery marketplaces across the world, we need to turn a wide variety of data into valuable products. The data is coming in from robots (eg telemetry, routing decisions, ETAs), merchants and customers (with their apps, orders, offering, etc), and all operational aspects of the business (from brief remote operator tasks to global logistics of spare parts and robots).

The diversity of use cases is the key reason that has attracted us to the data mesh approach — we want to carry out the data work very close to the people producing or consuming the information. By following data mesh principles, we hope to fulfil our teams’ diverse data needs while keeping central oversight reasonably light.

As Starship is not on enterprise scale yet, it’s not practical for us to implement all aspects of a data mesh. Instead, we have settled on a simplified approach that makes sense for us now and puts us on the right path for the future.

Data products

Define what your data products are — each with an owner, interface, and users

Applying product thinking to our data is the foundation of the whole approach. We think of anything that exposes data for other users or processes as a data product. It can expose its data in any form: as a BI dashboard, a Kafka topic, a data warehouse view, a response from a predictive microservice, etc.

A simple example of a data product in Starship might be a BI dashboard for site leads to track their site’s business volume. A more elaborate example would be a self-serve pipeline for robot software engineers for sending any kind of driving information from robots into our data lake.

In any case, we don’t treat our data warehouse (actually a Databricks lakehouse) as a single product, but as a platform supporting a number of interconnected products. Such granular products are usually owned by the data scientists / engineers building and maintaining them, not dedicated product managers.

The product owner is expected to know who their users are and what needs they are solving with the product — and based on that, define and live up to the quality expectations for the product. Perhaps as a consequence, we have started paying more upfront attention to interfaces, components that are crucial for usability but laborious to modify.

Most importantly, understanding the users and the value each product is creating for them makes it much easier to prioritize between ideas. This is critical in a startup context where you need to move quickly and don’t have the time to make everything perfect.

Data domains

Group your data products into domains reflecting the organizational structure of the company

Before becoming aware of the data mesh model, we had been successfully using the format of lightly embedded data scientists for a while in Starship. Effectively, some key teams had a data team member working with them part-time — whatever that meant in any particular team.

We proceeded to define data domains in alignment with our organizational structure, this time being careful to cover every part of the company. After mapping data products to domains, we assigned a data team member to curate each domain. This person is responsible for looking after the whole set of data products in the domain — some of which are owned by the same person, some by other engineers in the domain team, or even some by other data team members (e.g. for resource reasons).

There are a number of things we like about our domain setup. First and foremost, now every area in the company has a person looking after its data architecture. Given the subtleties inherent in every domain, this is possible only because we have divided up the work.

Creating structure into our data products and interfaces has also helped us to make better sense of our data world. For example, in a situation with more domains than data team members (currently 19 vs 7), we are now doing a better job at making sure each one of us is working on an interrelated set of topics. And we now understand that to alleviate growing pains, we should minimize the number of interfaces that are used across domain boundaries.

Finally, a more subtle bonus of using data domains: we now feel that we have a recipe for tackling all kinds of new situations. Whenever a new initiative comes up, it’s much clearer to everyone where it belongs and who should run with it.

There are also some open questions. Whereas some domains lean naturally towards mostly exposing source data and others towards consuming and transforming it, there are some that have a fair amount of both. Should we split these up when they grow too big? Or should we have subdomains within bigger ones? We’ll need to make these decisions down the road.

Data platform

Empower the people building your data products by standardizing without centralizing

The goal of the data platform in Starship is straightforward: make it possible for a single data person (usually a data scientist) to take care of a domain end-to-end, i.e. to keep the central data platform team out of the day-to-day work. That requires providing the domain engineers and data scientists with good tooling and standard building blocks for their data products.

Does it mean that you need a full data platform team for the data mesh approach? Not really. Our data platform team consists of a single data platform engineer, who is in parallel spending half of their time embedded into a domain. The main reason why we can be so lean in data platform engineering is the choice of Spark+Databricks as the core of our data platform. Our previous, more traditional data warehouse architecture placed a significant data engineering overhead on us due to the diversity of our data domains.

We have found it useful to make a clear distinction in the data stack between the components that are part of the platform vs everything else. Some examples of what we provide to domain teams as part of our data platform:

  • Databricks+Spark as a working environment and a versatile compute platform;
  • one-liner functions for data ingestion, e.g. from Mongo collections or Kafka topics;
  • an Airflow instance for scheduling data pipelines;
  • templates for building and deploying predictive models as microservices;
  • cost tracking of data products;
  • BI & visualization tools.

As a general approach, our aim is to standardize as much as it makes sense in our current context — even bits that we know won’t remain standardized forever. As long as it helps productivity right now, and doesn’t centralize any part of the process, we’re happy. And of course, some elements are completely missing from the platform currently. For example, tooling for data quality assurance, data discovery, and data lineage are things we have left for the future.

Data governance

Strong personal ownership supported by feedback loops

Having fewer people and teams is actually an asset in some aspects of governance, e.g. it is much easier to make decisions. On the other hand, our key governance question is also a direct consequence of our size. If there’s a single data person per domain, they can’t be expected to be an expert in every potential technical aspect. However, they are the only person with a detailed understanding of their domain. How do we maximize the chances of them making good choices within their domain?

Our answer: via a culture of ownership, discussion, and feedback within the team. We have borrowed liberally from the management philosophy in Netflix and cultivated the following:

  • personal responsibility for the outcome (of one’s products and domains);
  • seeking different opinions before making decisions, especially those impacting other domains;
  • soliciting feedback and code reviews both as a quality mechanism and an opportunity for personal growth.

We have also made a couple of specific agreements on how we approach quality, written down our best practices (including naming conventions), etc. But we believe good feedback loops are the key ingredient for turning the guidelines into reality.

These principles apply also outside the “building” work of our data team — which is what has been the focus of this blog post. Obviously, there is much more than providing data products to how our data scientists are creating value in the company.

A final thought on governance — we will keep iterating on our ways of working. There will never be a single “best” way of doing things and we know we need to adapt over time.

Final words

This is it! These were the 4 core data mesh concepts as applied in Starship. As you can see, we have found an approach to the data mesh that suits us as a nimble growth-stage company. If it sounds appealing in your context, I hope that reading about our experience has been helpful.

If you’d like to pitch in to our work, see our careers page for a list of open positions. Or check out our Youtube channel to learn more about our world-leading robotic delivery service.

Reach out to me if you have any questions or thoughts and let’s learn from each other!

[ad_2]

Source link

TagsarchitecturebestBusinesscoupledayforevergoodhappylakelightlivememodelnewperfecttrainingviewworkworldyoutube
Previous Article

Starship Completes One Million Autonomous Deliveries | ...

Next Article

10 secrets to building a successful digital ...

0
Shares
  • 0
  • +
  • 0
  • 0
  • 0

Related articles More from author

  • Travel & Lifestyle

    Faculty of Teacher Education Wins Sears Helgos Special Education Award

    August 23, 2022
    By admin1
  • All

    Convert Martech stack to CX stack

    August 24, 2022
    By admin1
  • Science & Tech

    Recognized for excellence in science and innovation in Waikato and the Bay of Plenty

    September 27, 2022
    By admin1
  • All

    SBS powers new digital marketing division

    August 18, 2022
    By admin1
  • Travel & Lifestyle

    Latest AI Announcements Mean Another Big Adjustment for Educators

    June 6, 2024
    By admin1
  • All

    Digital marketing holds the key to sales generation and economic recovery

    August 10, 2022
    By admin1

Leave a reply Cancel reply

You may interested

  • Science & Tech

    NSF funds new science teacher pipeline.news center

  • Science & Tech

    Science fiction virtual ghost metaphor, explained

  • Science & Tech

    Three companies create satellite launch guides

Search

Categories

  • All (1,240)
  • Animal Food (19)
  • Books & Novels (21)
  • Buying Guides (32)
  • Buying Guides (22)
  • Donation and Services (7)
  • Export Test (21)
  • Fashion (1,711)
  • Fashis (1)
  • Fitness & Health (20)
  • Food & Drinks (16)
  • Gift Guides (52)
  • Gift Guides (24)
  • Health & Beauty (21)
  • Health & Beauty (1,592)
  • Home&Living (218)
  • Marketing & Safety Solutions (3)
  • Mobility & Lifestyle (10)
  • Movies (1)
  • Non classifié(e) (2)
  • Nutritional Innovation (4)
  • Photo Gifts (1)
  • Price Comparison (1)
  • Science & Tech (1,346)
  • Science & Tech (1)
  • Sports (27)
  • Technology (157)
  • Travel & Lifestyle (1,652)
  • Travel & Lifestyle (14)
logo

Goevry is not just another run-of-the-mill magazine; it's a transformative journey that transcends the boundaries of traditional fashion publications. Our team of passionate experts, seasoned fashionistas, and visionary writers collaborate to curate a diverse range of thought-provoking features that delve into the very essence of style, culture, and identity.

  • Recent

  • Popular

  • Vom Bildschirm zum Bildschirm: Memmo Bringt Promis Direkt Zu Dir

    By admin1
    July 18, 2025
  • Experience Baseball More Than Just a Game: Step Up to the Plate with New York ...

    By admin1
    July 16, 2025
  • A Homecoming Story, An Original Documentary Featuring Giannis Antetokounmpo

    By admin1
    January 17, 2024
  • The Latest Science on the Global Climate Crisis

    By admin1
    August 16, 2022

Follow us

  • DMCA
  • Privacy Policy
  • About Us
  • Contacts
©2024 Copyright Goevry | All Rights Reserved.
We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.OkPrivacy policy