21 Oct
ELife
Junagadh
Our Company:
A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global.
Vision: Reliable ground transportation services globally with all types of vehicles.
Mission: Empower high quality local fleets
Coorporate Culture: team first
- Partner-centric
- Team collaboration,
- Never "not my job", end-to-end ownership,
- Continuous learning and improvement,
- Hard-working and pragmatic,
- Don’t be a middleman,
- Result-driven
Job title: Crawler Engineer
A Crawler Engineer is primarily responsible for designing and developing web crawler systems to scrape, clean, and analyze data from various platforms.
This position requires a deep understanding of how web crawlers work, familiarity with common anti-crawling techniques and countermeasures, and the ability to handle large-scale data processing.
**Primary Responsibilities**
- Design and develop efficient web crawler systems to meet business data scraping requirements.
- Conduct scraping strategy analysis on target websites and formulate optimal scraping plans.
- Maintain and optimize existing crawler systems to improve data scraping speed and accuracy.
- Clean and process scraped data to ensure data quality and availability.
- Keep track of and research the latest crawler technologies and anti-crawling mechanisms to continuously enhance the performance of crawler systems.
**Requirements**
- 5 years experience in similar positions
- Bachelor's degree or above in Computer Science or a related field, with a solid foundation in computer science.
- Proficient in Python programming, familiar with commonly used crawler frameworks (such as Scrapy, PySpider, etc.)
and information extraction techniques (such as regular expressions).
- Familiar with HTTP/HTTPS protocols, Cookie mechanisms, and web scraping principles.
- Proficient in JavaScript, XPath.
- Proficient in databases such as MySQL and BigQuery.
- Have knowledge of common anti-crawling techniques and countermeasures, able to tackle various anti-crawling challenges.
- Experience in designing and developing distributed systems, familiar with multithreading, asynchronous programming, and other technologies.
- Have good problem-solving skills and a strong teamwork spirit, able to work under pressure.
- Preference will be given to candidates with experience in scraping data from large platforms and handling massive datasets.
▶️ Software Engineer
🖊️ ELife
📍 Junagadh