Java 处理图片翻转 orientation

Posted by wxianfeng Sat, 21 Jun 2014 08:22:00 GMT

首先要理解 exif orientation 概念。非常重要,这个Orientation值提供了想要正常观看图像时应该旋转的方式。

例如 iPhone 正常观看是 Home 在右,横拍

Java 自动翻转处理代码,借助 thumbnailator 库实现:

pom.xml

<dependency>
  <groupId>net.coobird</groupId>
  <artifactId>thumbnailator</artifactId>
  <version>[0.4, 0.5)</version>
</dependency>

核心代码:

public static BufferedImage getImageFromUrl(String url) throws Exception {
    try {
        HttpClient client = HttpClientUtil.getHttpClient();
        HttpGet get = new HttpGet(url);

        HttpResponse response = client.execute(get);
        if (response.getStatusLine().getStatusCode() != 200) {
            EntityUtils.consumeQuietly(response.getEntity());
        }

        // return ImageIO.read(response.getEntity().getContent());
        // auto rotate image orientation
        return Thumbnails.of(response.getEntity().getContent()).scale(1).asBufferedImage();
    } catch (IOException e) {
        throw new IOException("getImageFromUrl Error");
    }
}

Python iso8601 东八区时间

Posted by wxianfeng Wed, 28 May 2014 07:40:00 GMT
import datetime
import pytz

dt = datetime.datetime.now()
print(dt)  # 2013-12-04 15:38:58.910402 本地时间,不带时区
pst = pytz.timezone('Asia/Shanghai')
dt = pst.localize(dt)
dt.astimezone(pst)
print(dt)  # 2013-12-04 15:38:58.910402+08:00

PostgreSQL 设置字段自增

Posted by wxianfeng Thu, 10 Apr 2014 07:30:00 GMT

利用序列可以实现字段的自增。

pgAdmin 入口:
pgAdmin → Tools → Query Tool

CREATE SEQUENCE users_id_seq
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;



alter table users alter column id set default nextval(users_id_seq);

查看表的创建脚本可以看到使用的序列:
pgAdmin → Object → SCRIPTS → Create Script


目标检测 IOU 计算

Posted by wxianfeng Mon, 03 Mar 2014 07:13:00 GMT

iou 为面积的交并比,在目标检测中非常常见,英文为 Intersection-over-Union。

图示:

计算格式:

python 代码:

class IOU:

    def calc_iou(self, bbox1, bbox2):
        """
        calc iou
        :param bbox1: (x1,y1,x2,y2)
        :param bbox2: (x1,y1,x2,y2)
        :return:
        """
        w_1 = abs(bbox1[0] - bbox1[2])
        h_1 = abs(bbox1[1] - bbox1[3])
        area_1 = w_1 * h_1

        w_2 = abs(bbox2[0] - bbox2[2])
        h_2 = abs(bbox2[1] - bbox2[3])
        area_2 = w_2 * h_2

        min_x_1 = min(bbox1[0], bbox1[2])
        min_y_1 = min(bbox1[1], bbox1[3])
        max_x_1 = max(bbox1[0], bbox1[2])
        max_y_1 = max(bbox1[1], bbox1[3])

        min_x_2 = min(bbox2[0], bbox2[2])
        min_y_2 = min(bbox2[1], bbox2[3])
        max_x_2 = max(bbox2[0], bbox2[2])
        max_y_2 = max(bbox2[1], bbox2[3])

        overlap_min_x = max(min_x_1, min_x_2)
        overlap_min_y = max(min_y_1, min_y_2)
        overlap_max_x = min(max_x_1, max_x_2)
        overlap_max_y = min(max_y_1, max_y_2)

        overlap_w = max(0, overlap_max_x - overlap_min_x)
        overlap_h = max(0, overlap_max_y - overlap_min_y)
        overlap_area = overlap_w * overlap_h

        iou = overlap_area / (area_1 + area_2 - overlap_area)

        return iou

Python 保存 Base64 图片到硬盘

Posted by wxianfeng Fri, 14 Feb 2014 12:53:00 GMT

服务端收到 base64 编码的图片,需要保存到硬盘。

注意需要先把 base64 字符串转为字节,然后使用 base64 的 decodebytes 解出。

核心代码:

with open(tmp_path, "wb") as fh:
    fh.write(base64.decodebytes(base64_str.encode()))

字符串和字节互转代码

1. To convert a string to bytes.
data = ""                    #string
data = "".encode()        #bytes
data = b""                     #bytes


2. To convert bytes to a String.
data = b""            #bytes
data = b"".decode() #string
data = str(b"")         #string


深度学习中 BoundingBox 坐标和 Rect 坐标区别

Posted by wxianfeng Thu, 02 Jan 2014 12:34:00 GMT

学习深度学习有一段时间了,大概一年,也算入门了,后面有时间慢慢给大家介绍。

今天给大家介绍下 BoundingBox 坐标和 Rect 坐标的区别。 ​ BoundingBox

BoundingBox 在目标检测(Object Detecting) 领域很常见,刚开始的时候,一直以为 (0,0) 坐标原点在左下角,正确的应该是在左上角,如下图所示:

BoundingBox 可简写为 bbox,有两个坐标 (xmin, ymin) (xmax, ymax),即左上角 和 右下角的两个点,对应图中的 (x1, y1) (x2, y2) 。

Rect

Rect 坐标是只有一个坐标点,然后有宽度(width),高度(height)的值,如下图所示:

这两种矩形标示法在机器学习领域都经常见到,没有好坏之分,殊途同归,且可以互相转化,如果设计 API 的话,个人认为选哪个都可以。

很简单,明白了吧 ~

BTW: 这篇文章是深度学习相关的第一篇文章,也是我第一次在微信公众号中写文章。这个微信公众号申请很久了,在2013年5月就申请了,还可以看到一个推送的表情 。为什么在微信公众号中写,以后 Blog 中不写了吗 ?Blog 中还会写,在微信公众号里写,1是为了增加文章的分发渠道,2是为了熟悉下微信公众号,因为后面还准备增加小程序。


使用 jstack 对 Java 线程 Dump 分析

Posted by wxianfeng Sun, 01 Dec 2013 03:13:00 GMT

用 httpclient 写了个代理服务, 没跑一会就会出现 504 TimeOut 错误, tomcat 假死现象, 业务日志无任何输出, 切入点使用 jstack 分析线程堆栈

jstack dump 堆栈

查看 Tomcat 进程

➜  photostore git:(photostore_1.0.0) ps aux | grep tomcat
haomiao         40556   0.0  0.6  6352212  51224   ??  S     6:35PM   0:38.59

jstack Dump 进程堆栈:

➜  photostore git:(photostore_1.0.0) jstack -l 40556  > /tmp/ps.log
"http-bio-8080-exec-8" #85 daemon prio=5 os_prio=31 tid=0x00007fb8e18e8000 nid=0x11a13 waiting on condition [0x000000011d20a000]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x000000079e359ad8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
    at org.apache.http.pool.PoolEntryFuture.await(PoolEntryFuture.java:138)
    at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306)
    at org.apache.http.pool.AbstractConnPool.access$000(AbstractConnPool.java:64)
    at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:192)
    at org.apache.http.pool.AbstractConnPool$2.getPoolEntry(AbstractConnPool.java:185)
    at org.apache.http.pool.PoolEntryFuture.get(PoolEntryFuture.java:107)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:276)
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:263)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at com.yunos.photostore.gatedlaunch.biz.http.HttpReuqestExecutor.doPOST(HttpReuqestExecutor.java:42)
    at com.yunos.photostore.gatedlaunch.web.MyAsyncServlet.service(MyAsyncServlet.java:72)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at com.taobao.tomcat.valves.ContextLoadFilterValve$FilterChainAdapter.doFilter(ContextLoadFilterValve.java:191)
    at com.taobao.eagleeye.EagleEyeFilter.doFilter(EagleEyeFilter.java:81)
    at com.taobao.tomcat.valves.ContextLoadFilterValve.invoke(ContextLoadFilterValve.java:150)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:429)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1085)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    - locked <0x000000079cff4ba0> (a org.apache.tomcat.util.net.SocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
    - <0x000000079cfec500> (a java.util.concurrent.ThreadPoolExecutor$Worker)

可以看到 线程都在 java.lang.Thread.State: WAITING (parking) 再看堆栈是在 httpclient 抛出来的, at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:306) 可以看出来 取连接的时候, 都被 Block 住了

netstat 看 TCP 连接, 果然存在 50 个 CLOSE_WAIT 的 TCP 连接

➜  photostore-test git:(master) ✗ netstat -a | grep rest.kanbox.com
tcp4       0      0  30.11.194.152.55421    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55418    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55402    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55400    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55398    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55396    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55394    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55392    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55390    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55388    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55386    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55384    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55382    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55380    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55378    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55376    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55374    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55372    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55370    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55368    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55366    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55364    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55362    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55360    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55357    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55338    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55336    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55127    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55090    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55089    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55088    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55087    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55086    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55085    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55084    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55081    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55079    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55075    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55058    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55050    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55025    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55021    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55020    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55019    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55017    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55016    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.55012    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.54985    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.54981    rest.kanbox.com.http   CLOSE_WAIT
tcp4       0      0  30.11.194.152.54980    rest.kanbox.com.http   CLOSE_WAIT

设置的 连接池 50 个, 所以只能处理 50 次请求就会出现 504 timeout 了, 因为连接池中取不到链接了。

CLOSE_WAIT 说明被动关闭了, 即对方关闭了连接, 自己没有关闭

解决办法

主动关闭连接: HttpClient 使用我们常用的InputStream.close()来确认连接关闭, httpclient 拿到 response, response 读取完了, 然后关闭流

public static void prepareForResponse(HttpServletResponse response, HttpResponse result) throws IOException {
    response.setStatus(result.getStatusLine().getStatusCode());
    response.setHeader("Content-Type","text/html;charset=UTF-8");
    InputStream inputStream = result.getEntity().getContent();
    OutputStream outputStream = response.getOutputStream();
    byte[] buffer = new byte[1024];
    int len ;
    while ((len = inputStream.read(buffer)) != -1) {
        outputStream.write(buffer, 0, len);
    }
    inputStream.close();
}

这个 case 告诉我们, Java 服务遇到假死的情况, 可以使用 jstack 神器。


ngx_lua 连接mysql

Posted by wxianfeng Sat, 16 Nov 2013 04:02:00 GMT

openresty 安装后, 默认包含 lua-resty-mysql, 我就是用的这个连接的 mysql, 很好用.

当一个 request 进来过后, 一般这么用:

local mysql = require "resty.mysql"
local db = mysql:new()
local cfg = { ... }
db:connect(cfg)
local res = db:query("select * from users")
ngx.say(json.encode(res))

但是如果我 require 的一个文件中也需要连接 mysql, 怎么办?

刚开始我这么用的:

a.lua

local Email = require("email")
local item = Email:get_item(5)

email.lua

local mysql = require("resty.mysql")
local db = mysql:new
db:connect(cfg)

function Email:get_item(id)
  local res = db:query("select * from emails where id = " .. id)
  local item = res[1]
  return item
end

一个 request 过来后, 直接报错:

> 2013/11/14 19:30:09 [error] 5290#0: *425 lua entry thread aborted: runtime
> error: attempt to yield across C-call boundary

出现这个原因是因为 require 的文件中, 不能直接有 mysql io 操作.

看下 春哥 的回复:

Hello!

2013/11/14 wxianfeng:
> 最新测试, 发现同时在两个lua文件中 db:connect(cfg) 不成功, 报下面错误:
>
> 2013/11/14 19:30:09 [error] 5290#0: *425 lua entry thread aborted: runtime
> error: attempt to yield across C-call boundary
>

这个错误是说你使用的某个 C 函数实现的 Lua 原语(比如 require 和 loadfile 之类)直接触发了 resty.mysql
的 I/O 操作(例如 connect 之类)。你应当避免让 require 和 loadfile 等操作直接触发 mysql 的 I/O
操作(比如把 mysql 相关的操作都放在你自己的 Lua 函数里,而不是放在 Lua 模块文件(.lua 文件)的顶层上)。

最后我把 email.lua 中连接 mysql 的操作, 放到一个函数中 解决.

local mysql = require("resty.mysql")
local db = mysql:new


local function connect_db()
  db:connect(cfg)
end

function Email:get_item(id)
  connect_db()
  local res = db:query("select * from emails where id = " .. id)
  local item = res[1]
  return item
end