mkdir SearchEngine
cd SearchEngine
git clone https://github.com/searxng/searxng-docker.git
git clone https://github.com/firecrawl/firecrawl.git

올인원 docker-compose.yml 없나?

cd searxng-docker
docker-compose up -d

✘ Container redis - Error response from daemon: Conflict. The container name "/redis" is already in use by container "7b1b566f173259879061f33f52ef94a5668cb7b2f85f0a39f89f7bdf9b132f5d". You have to remove (or rename) that container to be able to reuse that name.  

접두사를 붙혀서 문제 해결

이후 http://localhost:8080 에 접속해서 SearXNG 잘 동작하는지 확인하기

cd ..
cd firecrawl
docker compose build

https://docs.firecrawl.dev/contributing/self-host

아래와 같이 firecrawl 루트 디렉터리에 .env 만들기

# ===== Required ENVS ======
PORT=3002
HOST=0.0.0.0

# Note: PORT is used by both the main API server and worker liveness check endpoint

# To turn on DB authentication, you need to set up Supabase.
USE_DB_AUTHENTICATION=false

# ===== Optional ENVS ======

## === AI features (JSON format on scrape, /extract API) ===
# Provide your OpenAI API key here to enable AI features
# OPENAI_API_KEY=

# Experimental: Use Ollama
# OLLAMA_BASE_URL=http://localhost:11434/api
# MODEL_NAME=deepseek-r1:7b
# MODEL_EMBEDDING_NAME=nomic-embed-text

# Experimental: Use any OpenAI-compatible API
# OPENAI_BASE_URL=https://example.com/v1
# OPENAI_API_KEY=

## === Proxy ===
# PROXY_SERVER can be a full URL (e.g. http://0.1.2.3:1234) or just an IP and port combo (e.g. 0.1.2.3:1234)
# Do not uncomment PROXY_USERNAME and PROXY_PASSWORD if your proxy is unauthenticated
# PROXY_SERVER=
# PROXY_USERNAME=
# PROXY_PASSWORD=

## === /search API ===
# By default, the /search API will use Google search.

# You can specify a SearXNG server with the JSON format enabled, if you'd like to use that instead of direct Google.
# You can also customize the engines and categories parameters, but the defaults should also work just fine.
SEARXNG_ENDPOINT=http://host.docker.internal:8080
# SEARXNG_ENGINES=
# SEARXNG_CATEGORIES=

## === Other ===

# Supabase Setup (used to support DB authentication, advanced logging, etc.)
# SUPABASE_ANON_TOKEN=
# SUPABASE_URL=
# SUPABASE_SERVICE_TOKEN=

# Use if you've set up authentication and want to test with a real API key
# TEST_API_KEY=

# This key lets you access the queue admin panel. Change this if your deployment is publicly accessible.
BULL_AUTH_KEY=CHANGEME

# This is now autoconfigured by the docker-compose.yaml. You shouldn't need to set it.
# PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/scrape
# REDIS_URL=redis://redis:6379
# REDIS_RATE_LIMIT_URL=redis://redis:6379

# Set if you have a llamaparse key you'd like to use to parse pdfs
# LLAMAPARSE_API_KEY=

# Set if you'd like to send server health status messages to Slack
# SLACK_WEBHOOK_URL=

# Set if you'd like to send posthog events like job logs
# POSTHOG_API_KEY=
# POSTHOG_HOST=

## === System Resource Configuration ===
# Maximum CPU usage threshold (0.0-1.0). Worker will reject new jobs when CPU usage exceeds this value.
# Default: 0.8 (80%)
# MAX_CPU=0.8

# Maximum RAM usage threshold (0.0-1.0). Worker will reject new jobs when memory usage exceeds this value.
# Default: 0.8 (80%)
# MAX_RAM=0.8

docker-compose up -d

이렇게 뜨면 1차 관문 끝

아래 코드 날려보기. 잘 동작하면 OK

npm i @mendable/firecrawl-js

import Firecrawl from "@mendable/firecrawl-js";

const app = new Firecrawl({ apiUrl: "http://localhost:3002" });

// Scrape a website
const scrapeResponse = await app.scrapeUrl("https://firecrawl.dev", {
  formats: ["markdown", "html"],
});
console.log(scrapeResponse);

// Crawl a website (waiter)
const crawlResponse = await app.crawlUrl("https://firecrawl.dev", {
  limit: 100,
  scrapeOptions: { formats: ["markdown", "html"] },
});
console.log(crawlResponse);

이렇게도 쓸 수 있다

import Firecrawl from "@mendable/firecrawl-js";

const app = new Firecrawl({ apiUrl: "http://localhost:3002" });

app.search("JLPT와 JPT의 차이").then((response) => {
  console.log(response);
});

{
  success: true,
  data: [
    {
      url: 'https://namu.wiki/w/JPT',
      title: 'JPT - 나무위키',
      description: 'JLPT는 일본 정부가 직접 주관하는 시험으로 언어지식을 비롯한 실력을 보지만, JPT는 사설 시험이므로 주로 문법과 오문장 판단력 등등 일본어 디테일에 초점을 둔 시험  이라고 볼 수 있다.'
    },
    {
      url: 'https://blog.naver.com/witnwisdom/221239547255?viewType=pc',
      title: '대표적인 일본어 자격증 JLPT vs JPT 차이점은? : 네이버 블로그',
      description: 'Mar 28, 2018   ·   일본 유학, 해외 취업에 있어 보통 JLPT N1 ~ N2 레벨이 필요합니다. 반면 JPT는 등급 구분 없이 990점 만점을 기준으로 하는 점수제 시험이며,'
    },
    {
      url: 'https://www.amazingtalker.co.kr/blog/ko/new_japanese/75019/',
      title: 'JLPT와 JPT의 차이를 아시나요? 일본어 자격증 한눈에 알아보기!',
      description: 'Nov 10, 2022   ·   JLPT는 일본국제교육협회가 주관하는 시험이고, JPT는 한국 사설 기관에서 주관하는 시험입니다. JLPT는 1년에 단 2번만 시험 시행되는데요.'
    },
    {
      url: 'https://m.blog.naver.com/junhyuk7272/223029375884',
      title: '일본어 시험 JPT와 JLPT의 차이점과 일본어자격증 총정리 - 블로그',
      description: 'Feb 27, 2023   ·   JLPT 일본어 시험은 연간 2회만 시행되고 있으며 언어지식, 독해, 청해 파트별로 점수에 따라 N1~N5로 레벨이 나누어져 있습니다.'
    },
    {
      url: 'https://blog.class101.net/%EC%9D%BC%EB%B3%B8%EC%96%B4-%EB%8F%85%ED%95%99%EB%9F%AC%EB%A5%BC-%EC%9C%84%ED%95%9C-jpt-jlpt-%EB%B9%84%EA%B5%90-2024-jpt-%EC%8B%9C%ED%97%98%EC%9D%BC%EC%A0%95%EA%B9%8C%EC%A7%80/',
      title: '일본어 독학러를 위한 JPT JLPT 비교 2024 JPT 시험일정까지',
      description: 'May 10, 2024   ·   JPT와 JLPT 시험은 채점 · 급수합격 방식과 시험체계 및 구성 상의 차이점이 존재하며, 아래 점수 환산표는 JPT활용에 객관적 자료 제공을 목적으로, 두 ...'
    }
  ],
  warning: undefined
}

# You can specify a SearXNG server with the JSON format enabled, if you'd like to use that instead of direct Google.
# You can also customize the engines and categories parameters, but the defaults should also work just fine.
SEARXNG_ENDPOINT=http://localhost:8080
# SEARXNG_ENGINES=
# SEARXNG_CATEGORIES=

굳이 SearXNG를 로컬에 띄울 필요가 없어보인다;

searxng-docker > searxng > settings.xml에 아래 값을 넣어줘야 한다

search:
  formats:
    - html
    - json