Pinpoint-docker 설치하고 적용하기 [ agent 분리 ]

핀포인트는 (네이버에서) 만든 오픈소스 APM 도구로 매우 직관적이여서 만족도가 높았다. 이를 설치한 간단한 방법을 기록하려고 한다.

사실 매우 간단하다. 호스트에 Pinpoint를 직접 설치해서 사용할 때는 Hbase, Mysql, pinpoint collertor, pinpoint web 등 설치할 게 많아서 귀찮았는데 docker, 정확히는 docker-compose를 활용하면 이미 만들어진 공식 파일을 활용하면 된다. 

 

https://github.com/pinpoint-apm/pinpoint-docker

GitHub - pinpoint-apm/pinpoint-docker: Official Dockerized components of the Pinpoint

해당 깃헙주소를 참고하자면 Pinpoint v2.5.0부터 Url Metric과 Infrastructure Metric이 추가된 docker-compose-metrics.yml이 있는데 필자는 기본 docker-compose.yml을 활용하여 APM만 확인하도록 하겠다. 

 

docker-compose.yml

services:
  pinpoint-hbase:
    container_name: "${PINPOINT_HBASE_NAME}"
    image: "pinpointdocker/pinpoint-hbase:${PINPOINT_VERSION}"
    networks:
      - pinpoint
    environment:
      - AGENTINFO_TTL=${AGENTINFO_TTL}
      - AGENTSTATV2_TTL=${AGENTSTATV2_TTL}
      - APPSTATAGGRE_TTL=${APPSTATAGGRE_TTL}
      - APPINDEX_TTL=${APPINDEX_TTL}
      - AGENTLIFECYCLE_TTL=${AGENTLIFECYCLE_TTL}
      - AGENTEVENT_TTL=${AGENTEVENT_TTL}
      - STRINGMETADATA_TTL=${STRINGMETADATA_TTL}
      - APIMETADATA_TTL=${APIMETADATA_TTL}
      - SQLMETADATA_TTL=${SQLMETADATA_TTL}
      - TRACEV2_TTL=${TRACEV2_TTL}
      - APPTRACEINDEX_TTL=${APPTRACEINDEX_TTL}
      - APPMAPSTATCALLERV2_TTL=${APPMAPSTATCALLERV2_TTL}
      - APPMAPSTATCALLEV2_TTL=${APPMAPSTATCALLEV2_TTL}
      - APPMAPSTATSELFV2_TTL=${APPMAPSTATSELFV2_TTL}
      - HOSTAPPMAPV2_TTL=${HOSTAPPMAPV2_TTL}
    volumes:
      - hbase_data:/home/pinpoint/hbase
      - /home/pinpoint/zookeeper
    expose:
      # HBase Master API port
      - "60000"
      # HBase Master Web UI
      - "16010"
      # Regionserver API port
      - "60020"
      # HBase Regionserver web UI
      - "16030"
    ports:
      - "60000:60000"
      - "16010:16010"
      - "60020:60020"
      - "16030:16030"
    restart: always
    depends_on:
      - zoo1

  pinpoint-mysql:
    container_name: pinpoint-mysql
    image: mysql:8.0
    restart: "no"
    hostname: pinpoint-mysql
    entrypoint: > 
      sh -c "
      curl -SL "https://raw.githubusercontent.com/pinpoint-apm/pinpoint/${PINPOINT_VERSION}/web/src/main/resources/sql/CreateTableStatement-mysql.sql" -o /docker-entrypoint-initdb.d/CreateTableStatement-mysql.sql &&
      curl -SL "https://raw.githubusercontent.com/pinpoint-apm/pinpoint/${PINPOINT_VERSION}/web/src/main/resources/sql/SpringBatchJobRepositorySchema-mysql.sql" -o /docker-entrypoint-initdb.d/SpringBatchJobRepositorySchema-mysql.sql &&
      sed -i '/^--/d' /docker-entrypoint-initdb.d/CreateTableStatement-mysql.sql &&
      sed -i '/^--/d' /docker-entrypoint-initdb.d/SpringBatchJobRepositorySchema-mysql.sql &&
      docker-entrypoint.sh mysqld
      "
    ports:
      - "3306:3306"
    environment:
      - PINPOINT_VERSION=${PINPOINT_VERSION}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=${MYSQL_DATABASE}

    volumes:
      - mysql_data:/var/lib/mysql
    networks:
      - pinpoint

  pinpoint-web:
    container_name: "${PINPOINT_WEB_NAME}"
    image: "pinpointdocker/pinpoint-web:${PINPOINT_VERSION}"

    depends_on:
      - pinpoint-hbase
      - pinpoint-mysql
      - zoo1
      - redis
    restart: always
    expose:
      - "9997"
    ports:
      - "9997:9997"
      - "${WEB_SERVER_PORT:-8080}:8080"
    environment:
      - WEB_SERVER_PORT=${WEB_SERVER_PORT}
      - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES}
      - PINPOINT_ZOOKEEPER_ADDRESS=${PINPOINT_ZOOKEEPER_ADDRESS}
      - CLUSTER_ENABLE=${CLUSTER_ENABLE}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - CONFIG_SENDUSAGE=${CONFIG_SENDUSAGE}
      - LOGGING_LEVEL_ROOT=${WEB_LOGGING_LEVEL_ROOT}
      - CONFIG_SHOW_APPLICATIONSTAT=${CONFIG_SHOW_APPLICATIONSTAT}
      - JDBC_DRIVERCLASSNAME=${JDBC_DRIVERCLASSNAME}
      - JDBC_URL=${SPRING_DATASOURCE_HIKARI_JDBCURL}
      - JDBC_USERNAME=${SPRING_DATASOURCE_HIKARI_USERNAME}
      - JDBC_PASSWORD=${SPRING_DATASOURCE_HIKARI_PASSWORD}
      - SPRING_DATASOURCE_HIKARI_JDBCURL=${SPRING_DATASOURCE_HIKARI_JDBCURL}
      - SPRING_DATASOURCE_HIKARI_USERNAME=${SPRING_DATASOURCE_HIKARI_USERNAME}
      - SPRING_DATASOURCE_HIKARI_PASSWORD=${SPRING_DATASOURCE_HIKARI_PASSWORD}
      - SPRING_METADATASOURCE_HIKARI_JDBCURL=${SPRING_METADATASOURCE_HIKARI_JDBCURL}
      - SPRING_METADATASOURCE_HIKARI_USERNAME=${SPRING_METADATASOURCE_HIKARI_USERNAME}
      - SPRING_METADATASOURCE_HIKARI_PASSWORD=${SPRING_METADATASOURCE_HIKARI_PASSWORD}
      - SPRING_DATA_REDIS_HOST=${SPRING_DATA_REDIS_HOST}
      - SPRING_DATA_REDIS_PORT=${SPRING_DATA_REDIS_PORT}
      - SPRING_DATA_REDIS_USERNAME=${SPRING_DATA_REDIS_USERNAME}
      - SPRING_DATA_REDIS_PASSWORD=${SPRING_DATA_REDIS_PASSWORD}
      - PINPOINT_MODULES_WEB_LOGIN=${PINPOINT_MODULES_WEB_LOGIN}
      - WEB_SECURITY_AUTH_USER=${WEB_SECURITY_AUTH_USER}
      - WEB_SECURITY_AUTH_ADMIN=${WEB_SECURITY_AUTH_ADMIN}
      - WEB_SECURITY_AUTH_JWT_SECRETKEY=${WEB_SECURITY_AUTH_JWT_SECRETKEY}

    links:
      - "pinpoint-mysql:pinpoint-mysql"
    networks:
      - pinpoint

  pinpoint-collector:
    container_name: "${PINPOINT_COLLECTOR_NAME}"
    image: "pinpointdocker/pinpoint-collector:${PINPOINT_VERSION}"

    depends_on:
      - pinpoint-hbase
      - zoo1
      - redis
    restart: always
    expose:
      - "9991"
      - "9992"
      - "9993"
      - "9994"
      - "9995"
      - "9996"
    ports:
      - "${COLLECTOR_RECEIVER_GRPC_AGENT_PORT:-9991}:9991/tcp"
      - "${COLLECTOR_RECEIVER_GRPC_STAT_PORT:-9992}:9992/tcp"
      - "${COLLECTOR_RECEIVER_GRPC_SPAN_PORT:-9993}:9993/tcp"
      - "${COLLECTOR_RECEIVER_BASE_PORT:-9994}:9994"
      - "${COLLECTOR_RECEIVER_STAT_UDP_PORT:-9995}:9995/tcp"
      - "${COLLECTOR_RECEIVER_SPAN_UDP_PORT:-9996}:9996/tcp"
      - "${COLLECTOR_RECEIVER_STAT_UDP_PORT:-9995}:9995/udp"
      - "${COLLECTOR_RECEIVER_SPAN_UDP_PORT:-9996}:9996/udp"
    networks:
      pinpoint:
        ipv4_address: ${COLLECTOR_FIXED_IP}
    environment:
      - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES}
      - PINPOINT_ZOOKEEPER_ADDRESS=${PINPOINT_ZOOKEEPER_ADDRESS}
      - CLUSTER_ENABLE=${CLUSTER_ENABLE}
      - LOGGING_LEVEL_ROOT=${COLLECTOR_LOGGING_LEVEL_ROOT}
      - FLINK_CLUSTER_ENABLE=${FLINK_CLUSTER_ENABLE}
      - FLINK_CLUSTER_ZOOKEEPER_ADDRESS=${FLINK_CLUSTER_ZOOKEEPER_ADDRESS}
      - SPRING_DATA_REDIS_HOST=${SPRING_DATA_REDIS_HOST}
      - SPRING_DATA_REDIS_PORT=${SPRING_DATA_REDIS_PORT}
      - SPRING_DATA_REDIS_USERNAME=${SPRING_DATA_REDIS_USERNAME}
      - SPRING_DATA_REDIS_PASSWORD=${SPRING_DATA_REDIS_PASSWORD}

  pinpoint-quickstart:
    build:
      context: ./pinpoint-quickstart/
      dockerfile: Dockerfile

    container_name: "pinpoint-quickstart"
    image: "pinpointdocker/pinpoint-quickstart"
    ports:
      - "${APP_PORT:-8085}:8080"
    volumes:
      - data-volume:/pinpoint-agent
    environment:
      JAVA_OPTS: "-javaagent:/pinpoint-agent/pinpoint-bootstrap.jar -Dpinpoint.agentId=${AGENT_ID} -Dpinpoint.applicationName=${APP_NAME} -Dpinpoint.profiler.profiles.active=${SPRING_PROFILES}"
    networks:
      - pinpoint
    depends_on:
      - pinpoint-agent

  pinpoint-batch:
    container_name: "${PINPOINT_BATCH_NAME}"
    image: "pinpointdocker/pinpoint-batch:${PINPOINT_VERSION}"
    depends_on:
      - pinpoint-hbase
      - pinpoint-mysql
      - zoo1
    restart: always
    environment:
      - BATCH_SERVER_PORT=${BATCH_SERVER_PORT}
      - SPRING_PROFILES_ACTIVE=${SPRING_PROFILES}
      - PINPOINT_ZOOKEEPER_ADDRESS=${PINPOINT_ZOOKEEPER_ADDRESS}
      - CLUSTER_ENABLE=${CLUSTER_ENABLE}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD}
      - CONFIG_SENDUSAGE=${CONFIG_SENDUSAGE}
      - LOGGING_LEVEL_ROOT=${BATCH_LOGGING_LEVEL_ROOT}
      - CONFIG_SHOW_APPLICATIONSTAT=${CONFIG_SHOW_APPLICATIONSTAT}
      - BATCH_FLINK_SERVER=${BATCH_FLINK_SERVER}
      - JDBC_DRIVERCLASSNAME=${JDBC_DRIVERCLASSNAME}
      - JDBC_URL=${SPRING_DATASOURCE_HIKARI_JDBCURL}
      - JDBC_USERNAME=${SPRING_DATASOURCE_HIKARI_USERNAME}
      - JDBC_PASSWORD=${SPRING_DATASOURCE_HIKARI_PASSWORD}
      - SPRING_DATASOURCE_HIKARI_JDBCURL=${SPRING_DATASOURCE_HIKARI_JDBCURL}
      - SPRING_DATASOURCE_HIKARI_USERNAME=${SPRING_DATASOURCE_HIKARI_USERNAME}
      - SPRING_DATASOURCE_HIKARI_PASSWORD=${SPRING_DATASOURCE_HIKARI_PASSWORD}
      - SPRING_METADATASOURCE_HIKARI_JDBCURL=${SPRING_METADATASOURCE_HIKARI_JDBCURL}
      - SPRING_METADATASOURCE_HIKARI_USERNAME=${SPRING_METADATASOURCE_HIKARI_USERNAME}
      - SPRING_METADATASOURCE_HIKARI_PASSWORD=${SPRING_METADATASOURCE_HIKARI_PASSWORD}
      - ALARM_MAIL_SERVER_URL=${ALARM_MAIL_SERVER_URL}
      - ALARM_MAIL_SERVER_PORT=${ALARM_MAIL_SERVER_PORT}
      - ALARM_MAIL_SERVER_USERNAME=${ALARM_MAIL_SERVER_USERNAME}
      - ALARM_MAIL_SERVER_PASSWORD=${ALARM_MAIL_SERVER_PASSWORD}
      - ALARM_MAIL_SENDER_ADDRESS=${ALARM_MAIL_SENDER_ADDRESS}
      - ALARM_MAIL_TRANSPORT_PROTOCOL=${ALARM_MAIL_TRANSPORT_PROTOCOL}
      - ALARM_MAIL_SMTP_PORT=${ALARM_MAIL_SMTP_PORT}
      - ALARM_MAIL_SMTP_AUTH=${ALARM_MAIL_SMTP_AUTH}
      - ALARM_MAIL_SMTP_STARTTLS_ENABLE=${ALARM_MAIL_SMTP_STARTTLS_ENABLE}
      - ALARM_MAIL_SMTP_STARTTLS_REQUIRED=${ALARM_MAIL_SMTP_STARTTLS_REQUIRED}
      - ALARM_MAIL_DEBUG=${ALARM_MAIL_DEBUG}
    links:
      - "pinpoint-mysql:pinpoint-mysql"
    networks:
      - pinpoint

  pinpoint-agent:
    container_name: "${PINPOINT_AGENT_NAME}"
    image: "pinpointdocker/pinpoint-agent:${PINPOINT_VERSION}"
    restart: unless-stopped
    networks:
      - pinpoint
    volumes:
      - data-volume:/pinpoint-agent
    environment:
      - SPRING_PROFILES=${SPRING_PROFILES}
      - COLLECTOR_IP=${COLLECTOR_IP}
      - PROFILER_TRANSPORT_AGENT_COLLECTOR_PORT=${PROFILER_TRANSPORT_AGENT_COLLECTOR_PORT}
      - PROFILER_TRANSPORT_METADATA_COLLECTOR_PORT=${PROFILER_TRANSPORT_METADATA_COLLECTOR_PORT}
      - PROFILER_TRANSPORT_STAT_COLLECTOR_PORT=${PROFILER_TRANSPORT_STAT_COLLECTOR_PORT}
      - PROFILER_TRANSPORT_SPAN_COLLECTOR_PORT=${PROFILER_TRANSPORT_SPAN_COLLECTOR_PORT}
      - PROFILER_SAMPLING_TYPE=${PROFILER_SAMPLING_TYPE}
      - PROFILER_SAMPLING_COUNTING_SAMPLING_RATE=${PROFILER_SAMPLING_COUNTING_SAMPLING_RATE}
      - PROFILER_SAMPLING_PERCENT_SAMPLING_RATE=${PROFILER_SAMPLING_PERCENT_SAMPLING_RATE}
      - PROFILER_SAMPLING_NEW_THROUGHPUT=${PROFILER_SAMPLING_NEW_THROUGHPUT}
      - PROFILER_SAMPLING_CONTINUE_THROUGHPUT=${PROFILER_SAMPLING_CONTINUE_THROUGHPUT}
      - DEBUG_LEVEL=${AGENT_DEBUG_LEVEL}
      - PROFILER_TRANSPORT_MODULE=${PROFILER_TRANSPORT_MODULE}
    depends_on:
      - pinpoint-collector

  #zookeepers
  zoo1:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo1
    expose:
      - "2181"
      - "2888"
      - "3888"
    ports:
      - "2181"
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=0.0.0.0:2888:3888 server.2=zoo2:2888:3888 server.3=zoo3:2888:3888
    networks:
      - pinpoint

  zoo2:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo2
    expose:
      - "2181"
      - "2888"
      - "3888"
    ports:
      - "2181"
    environment:
      ZOO_MY_ID: 2
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=0.0.0.0:2888:3888 server.3=zoo3:2888:3888
    networks:
      - pinpoint

  zoo3:
    image: zookeeper:3.4.13
    restart: always
    hostname: zoo3
    expose:
      - "2181"
      - "2888"
      - "3888"
    ports:
      - "2181"
    environment:
      ZOO_MY_ID: 3
      ZOO_SERVERS: server.1=zoo1:2888:3888 server.2=zoo2:2888:3888 server.3=0.0.0.0:2888:3888
    networks:
      - pinpoint

  ##flink
  jobmanager:
    container_name: "${PINPOINT_FLINK_NAME}-jobmanager"
    image: "pinpointdocker/pinpoint-flink:${PINPOINT_VERSION}"
    expose:
      - "6123"
    ports:
      - "${FLINK_WEB_PORT:-8081}:8081"
    command: standalone-job -p 1 pinpoint-flink-job.jar -spring.profiles.active release
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
      - PINPOINT_ZOOKEEPER_ADDRESS=${PINPOINT_ZOOKEEPER_ADDRESS}
    networks:
      - pinpoint
    depends_on:
      - zoo1

  taskmanager:
    container_name: "${PINPOINT_FLINK_NAME}-taskmanager"
    image: "pinpointdocker/pinpoint-flink:${PINPOINT_VERSION}"
    expose:
      - "6121"
      - "6122"
      - "19994"
    ports:
      - "6121:6121"
      - "6122:6122"
      - "19994:19994"
    depends_on:
      - zoo1
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager
    networks:
      - pinpoint

  redis:
    image: redis:7.0.14
    restart: always
    hostname: pinpoint-redis
    ports:
      - "6379:6379"
    networks:
      - pinpoint

    
volumes:
  data-volume:
  mysql_data:
  hbase_data:

networks:
  pinpoint:
    driver: bridge
    ipam:
      config:
        - subnet: ${PINPOINT_NETWORK_SUBNET}

위 파일을 깃헙에서 복사해왔지만 pinpoint-quickstart는 삭제해도 좋다. 대신에 JVM 실행 명령에 다음과 같은 JAVA_OPT가 필요한 것은 확인하자.

JAVA_OPTS: "-javaagent:/pinpoint-agent/pinpoint-bootstrap.jar -Dpinpoint.agentId=${AGENT_ID} -Dpinpoint.applicationName=${APP_NAME} -Dpinpoint.profiler.profiles.active=${SPRING_PROFILES}"

필자는 pinpoint-agent도 필요가 없다. agent를 따로 다운받아서 도커파일에 넣어주었기 때문이다.

 

해당 주소에서 원하는 버전을 받을 수 있다. 하지만 2.5.0 이전은 web에서 쿼리가 안보이는 현상이 있어서 그 이상 버전을 받는 것을 추천한다.

wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.5~~~

tar -zxvf pinpoint-agent.tar.gz

https://github.com/pinpoint-apm/pinpoint/releases

Releases · pinpoint-apm/pinpoint

이후에 도커파일에 추가한 이 후 해당 JAVA_OPS를 추가하였다. 

-javaagent:./pinpoint-agent-2.5.3/pinpoint-bootstrap-2.5.3.jar -Dpinpoint.agentId=agentid -Dpinpoint.applicationName=application-name -Dpinpoint.config=./pinpoint-agent-2.5.3/pinpoint-root.config -Dprofiler.transport.grpc.collector.ip=colletor주소 -Dprofiler.server.excludeurl=/favicon.ico,/actuator/**

필자처럼 agent를 따로 구성할 때 colletor의 주소를 적어주자. 추가로 pinpoint-agent/pinpoint-root.conifg에 제외할 요청을 주거나 다른 옵션을 수정할 수 있으니 참고하면 좋을 듯 하다. -Dprofiler.server.excludeurl도 해당 config 파일에 동일하게 존재한다.

 

 

핀포인트의 간단한 아키텍쳐는 다음과 같다.

• Agent
  • 사용자 자바 애플리케이션이 특정 클래스를 불러오는 과정을 변조해서 해당 클래스의 행동 추적
  • 애플리케이션 프로세스의 데이터(memory, cpu 등)를 수집하여 collector로 전송
• Collector
  • Agent의 추적 정보를 받아 HBase에 저장
  • Agent가 측정한 프로세스 상태를 Flink로 전송
• Web
  • Collector가 HBase에 저장한 정보를 읽어 HTML로 렌더링

 

우리는 Pinpont-web을 통해서 확인하기 때문에 외부로 8080 port가 열려있어야하고 Agent가 수집한 정보를 Colletor가 받기 때문에 Agent가 실행되는(서비스가 실행되는) 환경으로부터 9991-9996 port가 열려있어야한다.

 

8080 port로 들어가 확인해보도록 하자.

 

추가로 운영 관점에서는 hbase를 분리하지 않는다면 보관주기를 꼭 설정하는 것이 관리하기 편할 것이다. 

환경 변수로 빼놓은 TTL들에 대해 각 운영 리스크에 맞게 설정하도록 하자.