Posted 3mo ago

Lead Observability Engineer

@ Staples
Chennai, Tamil Nadu, India
OnsiteFull Time
Responsibilities:Designing Splunk, Managing deployments, Building dashboards
Requirements Summary:6-8 years of experience in observability, Splunk Cloud administration, and data monitoring; strong automation and scripting skills; cloud experience.
Technical Tools Mentioned:Splunk Cloud, Terraform, Ansible, Puppet, Jenkins, Python, Bash, GCP, Azure, AWS
Save
Mark Applied
Hide Job
Report & Hide
Job Description
  • Splunk Cloud Administration & Strategy 

  • Design, implement, and manage Splunk Cloud environments to support scalable log ingestion, indexing, and search performance. 

  • Ensure high availability, data retention, and disaster recovery strategies are in place. 

  • Automation & Integration 

  • Automate configurations and deployments using tools like Terraform, Ansible, Puppet or custom scripts. 

  • Build and manage integrations with CI/CD pipelines, cloud-native services (GCP, Azure), ITIL platforms for real-time observability. 

  • Develop and maintain custom Splunk apps and modular inputs to extend platform capabilities. 

  • Monitoring & Optimization 

  • Monitor Splunk performance metrics and proactively optimize indexing, search performance, and storage usage. 

  • Tune data inputs, parsing rules, and indexing strategies to ensure efficient resource utilization. 

  • Implement best practices for data onboarding, including source types, field extractions, and event normalization. 

  • Security & Compliance 

  • Manage user roles, access controls, and authentication mechanisms (SSO, LDAP, SAML). 

  • Ensure compliance with data governance policies and audit requirements through secure logging practices. 

  • Collaboration & Leadership 

  • Partner with application teams, SREs, and DevOps to onboard new data sources and build meaningful dashboards. 

  • Conduct knowledge-sharing sessions and mentor junior team members on best practices. 

  • Participate in architectural reviews and provide guidance on observability strategies across teams. 

  • Continuous Improvement 

  • Stay current with latest features, releases, and ecosystem tools. 

  • Contribute to the evolution of logging and monitoring standards across the organization. 

  • Identify opportunities to improve operational efficiency through enhanced visibility and automation. 



Requirements

  • Splunk Certified Admin / Architect 

  • Proficient in scripting languages (Python, Bash, etc.) and automation tools (Puppet/Ansible/Terraform/Jenkins) 

  • Experience working within a SAFe environment, including participation in PI (Program Increment) Planning, Agile Release Trains (ARTs), and cross-functional collaboration across teams. 

  • Proficient in managing data inputs and universal forwarders 

  • Proficient in setting up users, roles, and authentication protocols to ensure secure access control 

  • Extensive knowledge of Splunk configuration files and their role in customizing deployments 

  • Experience in installation and management of apps within Splunk Cloud to extend functionality 

  • Skilled in monitoring, problem isolation, and system health checks to maintain performance 

  • Deep understanding of cloud platforms like Azure and GCP 

  • Demonstrated expertise in sizing, planning, and deploying distributed Splunk environments 

  • Familiarity with indexer and search head clustering 

  • Demonstrated expertise in onboarding diverse data sources and optimizing parsing for performance and scalability 

  • Experience integrating Splunk with other enterprise systems (CMDB, ticket tools, etc.) 

  • Creating intuitive, actionable dashboards according to best practices for Splunk performance