Data Engineer - Metering

DataDog


1 month ago

08/17/2019 10:21:23

Job type: Full-time

Hiring from: US only

Category: All others


About Datadog:

We're on a mission to build the best platform in the world for engineers to understand and scale their systems, applications, and teams.  We operate at high scale—trillions of data points per day—providing always-on alerting, metrics visualization, logs, and application tracing for tens of thousands of companies. Our engineering culture values pragmatism, honesty, and simplicity to solve hard problems the right way.

The team:

The Revenue and Growth Team builds and runs the data pipelines, container-native services, and systems to quantify our customers’ usage across all Datadog products. This team is at the leading edge of any new product we release.

The opportunity:

As a Data Engineer within the Revenue & Growth Metering team, you will work in Spark with big data tooling to build highly reliable, verifiably-accurate data processing pipelines for a high scale mission-critical process. This team ingests the full firehose of data we receive each day - literally trillions of data points and hundreds of TB’s.

You will:

  • Build distributed, high-volume data pipelines that power this core product

  • Do it with Spark, Luigi and other open-source technologies

  • Work all over the stack, moving fluidly between programming languages: Scala, Java, Python, Go, and more

  • Join a tightly knit team solving hard problems the right way

  • Own meaningful parts of our service, have an impact, grow with the company

Requirements:

  • You have a BS/MS/PhD in a scientific field or equivalent experience

  • You have built and operated data pipelines for real customers in production systems

  • You are fluent in several programming languages (JVM & otherwise)

  • You enjoy wrangling huge amounts of data and exploring new data sets

  • You value code simplicity and performance

  • You want to work in a fast, high growth startup environment that respects its engineers and customers

Bonus points:

  • You are deeply familiar with Spark and/or Hadoop

  • In addition to data pipelines, you’re also quite good with Chef or Puppet

  • You’ve built applications that run on AWS

  • You’ve built your own data pipelines from scratch, know what goes wrong, and have ideas for how to fix it

Please mention that you come from Remotive when applying for this job.

Help us maintain Remotive! If this link is broken, please just click to report dead link!

similar jobs

  • 2 days ago

    Do you enjoy working with data architecture, migrations, scripting, and servers? Are you enthusiastic to push the web forward, and better yourself? We are also passionate about the web here at WebDevStudios, so let’s chat.

    YOUR DAY-TO-DAY

    • You are primarily responsible for migrating data from one web application to another according to specifications outlined by the client and defined by our Technical Strategist during the Discovery Phase.

    • Data Engineers work alongside a Project Team and other partners during the entire development life cycle of a project, as well as a supporting the project after it ships.

    • Technologies you’ll be working with include: Docker and containerization, CI orchestration tools, Linux servers, Relational databases, PHP7, Composer, etc

    QUALITIES THAT MATTER TO OUR TEAM:

    • A positive attitude, a good work ethic, and enjoy helping others

    • Not afraid to ask for help, teachable, a quick study, and have the ability to be flexible when “stuff happens”

    • Understand deadlines and can complete a variety of tasks throughout the week

    • Willing to participate in company paid training courses to further their skills

    • Excited about contributing back to open-source technologies

    • Comfortable communicating via Slack and participating in video calls with their team while working remotely

    RESPONSIBILITIES

    • Migrate data from one web application to another (e.g., a Drupal to WordPress migration)

    • Maintain our development, CI, and deployment infrastructure

    • Optimize client production infrastructure for maximum speed, scalability, and maintainability

    • Meet documented deadlines and complete tasks in accordance to project milestones

    • Work directly with Technical Strategists, Project Managers, and Senior Engineers on data architecture and application infrastructure

    • Communicate the status of your work to Project Managers and participate in a project team and department calls

    • Keep an accurate record of your time with our time tracking software

    • Be available during normal business hours

    • Able to work remotely, with a strong emphasis on communication and organizational skills

    REQUIREMENTS

    • You have 2+ years professional experience as a Data or Backend Engineer

    • Expert level experience with data extraction, transformation and loading from various sources

    • Expert level experience with WordPress plugin and theme structure, template tags, custom post meta, internationalization, and escaping/sanitization of data

    • Expert level experience configuring and administrating relational databases like MySQL and MSSQL

    • Experience analyzing and architecting new and existing data and recommend improvements

    • Experience providing best practice guidance for data technologies

    • Experience with PHP/Bash/WP-CLI scripting to link, migrate, and process data

    • Experience with Jenkins/CircleCI/Buddy and other orchestration tools

    • Experience using AWS, Google Cloud, WPEngine, Docker and other services and tools

    • Experience with Algolia Search, ElasticSearch, Redis, or Memcache

    • Experience with local environment setup for web application development, including Docker

    • Experience with Git version control and branching strategies

    • Experience creating monitoring strategies for infrastructure

    • Ability to use a wide variety of open source technologies and tools

    EXTRA CREDIT

    It’s not required, but experience or interest in the following would be nice:

    • Can read, understand, and write modern PHP and JavaScript

    • Have worked on high profile WordPress web applications before

    • WordPress plugins like:

    • Advanced Custom Fields

    • Beaver Builder

    • CMB2

    • Gravity Forms

    • Yoast SEO

    • ADDITIONAL DETAILS

    • REST APIs, especially the WordPress REST API

    • Postman or other REST API tools

    • GraphQL

    • Dress code: pajamas, if you like.

    • This is a full-time, salaried position.

    • Full-time employees enjoy benefits such as Medical/Dental/Vision, Paid Time Off, 401K, Hardware stipend, and more!

    • Pay rate is based on experience and abilities.

    • Lots and lots of informal GIF sharing.

    • OH, THE BENEFITS

    We know the value in supporting our employees’ holistic well-being, and we make sure our benefits package reflects our commitment to our team.

    • 100% Paid Medical

    • Paid Sick Leave

    • Unlimited Flex Time

    • 25 days paid time off

    • Work from anywhere

    • Employee 401k

    • Continuing Education

    • Parental Leave

    • Annual Company Retreat

  • Zignal Labs (US only)
    6 days ago

    About Zignal Labs

    Zignal Labs is the world’s leading media analytics company, helping companies build and protect their most valuable asset: their brand. With unparalleled data veracity, speed to surface insights and a holistic view of the traditional and new media landscape, Zignal empowers the most innovative communications and marketing teams across the Fortune 1000 to measure the conversation around their brands in real-time, rapidly identify and mitigate reputational risks and inform strategic decision-making to achieve mission-critical business outcomes. Headquartered in San Francisco with offices in New York City and Washington D.C., Zignal serves customers around the world, including Expedia, GoPro, DaVita, Under Armour, Synchrony, Prudential, DTE Energy, The Public Goods Project and Uber. To learn more, visit: www.zignallabs.com.

    About the Role

    As a Data Scientist on our Labs and Data Science team, you will work on data analytics and machine learning projects and be involved in solutions from ideation through research and prototyping up to feature delivery-including data quality measurements and improvements. You will rely on your Scala and Python coding skills to analyze a large amount of media data and build machine learning models. You will use Spark, SageMaker, S3, and ElasticSearch along with your machine learning and NLP skills and apply it to social media, news, blogs, broadcast, and other media sources to empower our users with key insights based on real-time analysis.

    In this role, you will have the opportunity to:

    • Mine and analyze media data from various data sources to create and improve analytical insights, product and application features

    • Measure and improve the effectiveness and accuracy of new and existing features

    • Develop custom data models and algorithms to apply to data sets

    • Extend company quality measurement frameworks and test and improve model quality

    • Coordinate with different functional teams to implement models and monitor outcomes

    • Develop processes and tools to monitor and analyze model performance and data accuracy

    Tech Stack:

    • Scala, Python

    • Spark / Databricks

    • Nice to have: S3, Elasticsearch, Amazon Sagemaker, Amazon Mechanical Turk

    In order to be successful in this role, you will need:

    • Master's degree in Computer Science, Mathematics or equivalent field with 5+ years work experience, or

    • Bachelor's degree with 8+ years relevant experience

    • Strong problem-solving skills with an emphasis on product development

    • Excellent written and verbal communication skills for cross-team collaboration

    • Knowledge of a variety of machine learning techniques (clustering, decision tree learning, artificial neural networks, etc.) and their real-world advantages/drawbacks

    • A passion to learn and master new technologies and techniques

    • Coding knowledge and experience with Scala and/or Python

    • Experience with one or more of the following distributed data/computing tools: Spark, Map/Reduce, Hadoop, Hive

    • Familiarity with data quality measurement techniques, and processes

    Plusses:

    • Experience with various natural language processing techniques: parts of speech tagging, shallow parsing, constituency, and dependency parsing, named entity recognition, emotion and sentiment analysis

    • Experience building and using advanced machine learning algorithms and statistics: regression, simulation, scenario analysis, modeling, clustering, decision trees, neural networks, etc.

    • Intellectual curiosity around our business & tech challenge

    Why Join Zignal?

    • Competitive salary based on the work you do

    • 100% employer-paid Medical, Dental, and Vision insurance

    • Flexible time off – work with your manager to take the time you need

    • Subsidized commuter benefits

    • Up to 16 hours of paid time off to volunteer in your local community

    • Learning environment where we value professional and personal development

    • Catered lunches 3 times a week and fully stocked kitchen

    • Our office is located in the Financial District just blocks away from BART

    Applicants must be authorized to work in the United States for any employer. No sponsorship is available for this position now or in the future and or visa transfers now or in the future. Remote option available within U.S.

  • Mammoth Growth (US or Canada)
    3 weeks ago

    Mammoth Growth is seeking a Data Engineer with extensive experience in building data pipelines, ETL scripts, and data warehouses in a modern cloud environment. We are a fast-paced, rapidly growing growth data analytics consultancy helping businesses build cutting edge analytics environments.

    As a data engineer in a rapidly growing team, you will work with a variety of exciting high growth businesses building their future data environment. This is an excellent opportunity to sharpen and broaden your skills in a fast-paced, challenging environment.

    This is a remote position.

    Responsibilities

    • Build custom integrations with 3rd party APIs

    • Building ETLs to move and transform data

    • Put together an end-to-end data pipeline using cutting edge tools and techniques

    • Designing data warehouses and data lakes

    • Use your knowledge and experience to help shape new process

    Skills

    • Python

    • AWS Lambda

    • SQL

    • Spark / Databricks / AWS Glue

    • Database experience (Redshift, Snowflake or Data Lake a plus)

    Qualities

    • Independently organized; self-starter; ability to work with minimal direction

    • Enjoy learning new tools to better solve new challenges

    • Attention to detail and ability to ask the right questions

    • Good communication / client facing skills

    • Can switch between simultaneous projects easily

    If you think you are a good fit for the role send us a quick note on why and include the sum of 18 and 22 (bonus points for creativity).

Remotive can help!

Not sure how to apply properly to this job? Watch our live webinar « 3 Mistakes to Avoid When Looking For A Remote Startup Job (And What To Do Instead) ».

Interested to chat with Remote workers? Join our community!